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ABSTRACT 

This book reports on the 1994 National Assessment for 
Educational Progress (NAEP) assessment in geography for grades 4, 8, 
and 12* Students performance is summarized on a scale ranging from 0 
to 500 with the achievement levels of Basic, Proficient and Advanced 
for each grade* Major findings for the nation include: (1) The 
Proficient level was reached by 22 percent of fourth graders, 28 
percent of eighth graders, and 27 percent of twelfth graders; (2) At 
each graae level, roughly 70 percent of students were at or above the 
Basic level; (3) As students' geography scores increased, the 
complexity and sophistication of the geographic knowledge and skills 
they exhibited increased; and (4) Generally, students across grades 
in the higher percentiles exhibited greater abilities to work with a 
range of geographic tools, create maps based on tabular or narrative 
data, grasp processes and relationships, bring outside knowledge to 
bear on answering questions and analyze data* The book contains five 
chapters* Chapter 1, "NAEP 1994 Assessment in Geography," presents 
the overview of the NAEP 1994 geography assessment, including its 
content framework , design, and admin i s trat i on, along with sample 
questions and student responses from the assessment* Chapter 2, 
"Geography Results For the Nation and Regions," provides overall 
average scale score results for the nation, regions, and subgroups of 
students* Chapter 3, "Geography Achievement Levels," describes 
student performance in terms of achievement levels* Chapter 4, 
"Contexts in Which Students Learn Geography," describes contextual 
factors related to students' geography achievement* Chapter 5, "What 
Students Know and Can Do in Geography," describes the specific 
abilities that students demonstrated on the NAEP 1994 geography 
assessment and reports student performance in different content areas 
of geography* Three appendices and extensive tables and figures 
accompany the text . (EH) 
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THE NATION‘S REPORT CARD, the National Assessment of Educational Progress (NAEP), is the only nationally representative and continuing 
assessment of what America’s students know and can do in various subject areas. Since 1969, assessments have been conducted periodically in 
reading, mathematics, science, writing, history/geography, and other fields. By making objective information on student performance available to 
policymakers at the national, state, and local levels, NAEP is an integral part of our nation’s evaluation of the condition and progress of education. 
Only information related to academic achievement is collected under this program. NAEP guarantees the privacy of individual students and their 
families. 

NAEP is a congressionally mandated project of the National Center for Education Statistics, the U.S. Department of Education. The 
Commissioner of Education Statistics is responsible, by law, for carrying out the NAEP project through competitive awards to qualified organiza- 
tions. NAEP reports directly to the Commissioner, who is also responsible for providing continuing reviews, including validation studies and 
solicitation of public comment, on NAEP’s conduct and usefulness. 

In 1988' Congress established the National Assessment Governing Board (NAGB) to formulate policy guidelines for NAEP. The Board is 
responsible for selecting the subject areas to be assessed from among those included in the National Education Goals; for setting appropriate student 
performance levels; for developing assessment objectives and test specifications through a national consensus approach; for designing the 
assessment methodology; for developing guidelines for reporting and disseminating NAEP results; for developing standards and procedures for 
interstate, regional, and national comparisons: for determining the appropriateness of test items arid ensuring they are free from bias; and for taking 
actions to improve the form and use of the National Assessment. 
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If policymakers, educators, and concerned citizens are 
to reform and improve the United States educational 
system, they need valid and reliable information on the 
streng^s and weaknesses of American students and on 
the instructional factors that are related to differing 
levels of performance. For more than 25 years, the 
National Assessment of Educational Progress (NAEP) 
has provided such information. NAEP assessments have 
probed students’ abilities in a variety of subject areas, 
reporting both on what students know and can do ^d 
on the relationships between instructional, institutional, 
and background variables and differing levels of educa- 
tional achievement. As the nation’s foremost ongoing 
education survey, the national assessment data track 
trends in student performance and allow concerned 
readers to eveJuate whether America’s students have the 
skills and knowledge necessary to participate in today’s 
economic and political worlds. 

In 1994 NAEP conducted national assessments in 
reading, geography, and United States history at grades 
4, 8, and 12. The geography results included in this 
Report Card describe students’ achievement at each 
grade and within subgroups of the general population. 

In addition, the report discusses the relationships among 
student performance and instructional and home back- 
ground variables. Tbken together, this information Vdll 
give educators a context for evaluating the geography 
achievement of students and data that may be used to 
guide reform efforts. .'i,., 

Student performance oh the NAEP 1994 geography 
assessment is summarized on the NAEP geography scale, 

‘ which ranges from 0 to 500. The geography scale allow for 
the discussion of what students know and can do in terms 
of the geography content covered by the assessment In 
addition, results are reported according to geograi>hy 
achievement levels adopted by the National Assessment 
Governing Board. For each grade, three achievement 
levels were set — Basic, Proficient, and Advanced. These 
are based on judgments, made by broadly representative 
panels, about what students should know and should be 
able to do in geography. 

The Proficient achievement level represents solid 
academic performance that demonstrates competency 
over challenging subject matter for each grade assessed. 
The Basic achievement level denotes partial mastery of 
prerequisite knowledge and skills that are fundamental 
for proficient work. The Advanced achievement level 
signifies superior performance. 




To maximize usefulness to policymakers, educators, 
parents, and other inte rested parties, the NAEP results 
are presented both as average scores on the NAEP 
geography scale, and in terms of the percentage of 
students attaining NAEP geography achievement levels. 
Thus, NAEP results not only provide information about 
what students know and can do, but also indicate 
whether their achievement meets expectations of what 
students should know and should be able to do. Further- 
more, the descriptions of skills and abilities expected of 
students at each achievement level help make the 
reporting of assessment results more meaningftil. 

Major Findings for the Nation 

► The Proficient achievement level was reached by 
22 percent of fourth graders, 28 percent of eighth 
graders, and 27 percent of twelfth graders. 

► At each grade, roughly 70 percent of students were 
at or above the Basic level. 

► As students' geography scores increased, the 
complexity and sophistication of the geographic 
knowledge and skills they exhibited increased. 

For example: 

• At grade 4, 79 percent of students could identify 
the water cycle from an illustration; 70 percent 
could draw a generally accurate nap of an island 
from a written description; 59 percent could use a 
map to explain the concentration of Wghways in 
the eastern United States; and 13 percent could 

' describe two important effects of an oil spill in 
the ocean. ' ^ • .. ' 

• At grade 8, 90 percent of students knew where to 
- locate information in an atlas; 70 percent could 

understand why immigrants congregate in New 
York City; 48 percent could identify latitude on a 
polar map projection; and 36 percent could identify 
and e]q)lain two reasons why a particular route 
for a railroad would prove cheaper to construct 
than an alternate route. 

• At grade 12, 91 percent of students could use a 
map •’0 identify an area of earthquake activity; 

66 percent could construct a precipitation pie 
chart from tabled data; 55 percent could give at 
least two geographically accurate reasons that a 
shopping center should be placed at a given 
location; and 10 percent could identify Canada as 
the United States’ largest trading partner. 
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> Generally, students across grades in the higher 
percentiles exhibited greater abilities to work with 

;a range of geographic tools, create maps based on 
tabular or narrative data, grasp processes and 
relationships, bring outside knowledge to bear on 
answering questions, and analyze data. 

Major Findings for Student Subgroups 

> As in other NAEP assessments, statistically significant 
differences existed in the performance of major 
subgroups of the population. For example, at all 
three grades. White and Asian students had higher 
scores than did their Black and Hispanic counter- 
parts. In addition, at all grades, Hispanic students had 
higher average scale scores than did Black students. 

■> Consistent with findings in other assessments, there 
was a strong relationship between differing levels of 
parental education and performance in geography. 
As a general rule, the more education students’ 
parents had received, the better they performed on 
the assessment. 

^ On the overall geography scale, male students 
performed better than female students at all three 
grades. However, gender differences were not consis- 
tent across content areas within geography. For 
example, at grade 4 males outperformed females on 
tasks assessing the content area “Space and Place,” 
while there were no significant performance differ- 
ences in the two other content areas. 

^ At all three grades, students attending nonpublic 
_ schools performed at a higher level than did those 
i attending public schools. 
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Contextual Factors Related to 
Geography Performance 

A diverse range of home and school factors are related to 
the ways and extent to which students learn geography. 
Students who participated in the NAEP assessment were 
asked to complete questionnaires about home and 
school experiences related to geography learning. Also, 
teachers and school administrators completed question- 
naires about their students’ instructional e3q>eriences. 
'The results of these surveys help place the assessment 
scores into context, and allow policymakers to determine 
which variables are positively and negatively related to 
geography achievement. 

!>■ Over 40 percent of the students at grades 4 and 8, 
and 25 percent of the students at grade 12 reported 
watching four or more hours of television each day. 
In most cases, the more television students rejj^rted 
watching, the worse they performed on the geog- 
nq>hy assessment. 

► Fifty-sbc percent of fourth graders, 39 percent of 

eighth graders, and 31 percent of twel^ graders 
reported discussing their studies at home daily. By 
contrast, 17, 21, and 24 percent of students at 
grades 4, 8, and 12, respectively, reported never or 
hardly ever discussing their studies at home. Stu- 
dents who reported not discussing their ^dies at 
home performed at a lower level than (fid students 
who discussed their studies on a regular basis. , 

^ Geography instruction is limited for grade 4 students. 
More than 60 percent of students had tochers who 
reported spending less than 45 minulb^'per week on 
geography instruction. Most eighth-grade students 
'^reported having taken at least one geography class 
since the sixth grade. v 

► TWenty-six percent of fourth graders, 19 percent of 

eighth graders, and 14 percent of twelfth graders 
indicated that geography was their fevorite subject. 
At all grades, students who indicated tl»t geography 
was their favorite subject performed at ahigher level 
than did those who indicated that they liked other 
subjects better. , , 
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About This Report 

As the nation’s Report Card in geography, this 
document provides a broad examination of students’ 
learning. In addition, specific aspects of students’ 
performance and their experiences at home and school 
are reviewed in some depth. As such, this report 
provides a portrait of what students know and can do in 
geography, as well as the contexts in which they have 
developed their geographic knowledge and skills. 

A limited assessment of the geography achievement 
of high school seniors was conducted by NAEP in 
conjunction with the National Geographic Society in 
1988. However, the content framework that underlies 
the NAEP 1994 geography assessment is markedly 
different from the framework used for the 1984 
assessment. Therefore, information cannot be reported 
on trends in high school seniors’ geography achievement 
between 1988 and 1994. 

Chapter 1 presents the overview of the NAEP 1994 
geography assessment — its content framework, design, 
cmd administration. Also included in Chapter 1 are 
sample questions and student responses from the 
assessment. Chapter 2 provides overall average scale 
score results for the nation, regions, and subgroups of 
students. Chapter 3 describes student performance in 
terms of achievement levels. Chapter 4 describes 
contextual factors related to students’ geography 
achievement. Cluqjter 5 describes the specific abilities 
that students demonstrated on the NAEP 1994 
geography assessment and reports student performance 
in different content areas of geography. 
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NAEP 1994 Assessment 
in Geography 

“The social progress, order, security, and peace of each 
country are necessarily connected with the social 
progress, order, security, and peace of all other countries.” 

Pope John XXIII 

Introduction 

Governments, economies, and ecosystems do not exist 
in isolation. People and regions are connected by trade 
agreements, global markets, communications networks, 
political alliances, and international organizations. 
Increasingly, nations are sharing concerns about the 
global environment United States foreign and economic 
policies must account for events across the globe: 
Competition for jobs in Peoria is as likely to come from 
Kuala Lumpur as from Fresno. Events from around the 
world, such as the unification of Germany, afreet us in 
profound ways. If our children are to be productive and 
responsible citizens of both the United States and the 
global community, they must know and understand the 
connections among the world’s regions and peoples and 
the circumstances that lead these connections to evolve 
and change. In other words, they must have a working 
knowledge of the subject of geography. 

Geography in American Education 

In spite of the importance of geographic skills and 
competencies, geography has occupied an inconsistent 
place in the American classroom. In the nineteenth 
century, geography was viewed as a core element of the 
school curriculum. Howwer, with the growth of the 
held of social studies since the second decade of the 
twentieth century, geography began to be squeezed out 
of the curriculum. In addition, pressure on schools to 
include a range of difrerent subjects in their instructional 
programs has further reduced the time allocated to 
geography.* 

Attitudes toward the place of geography in the 
school curriculum began to undergo a substantial 
change in the 1980s. Concerned about levels of 
geographic literacy in our nation’s schools, the National 
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Council for Geographic Education and the American 
Association of Geographers created a joint committee to 
reestablish geography in the school curriculum. Their 
report. Guidelines for Geographic Education: Elementary 
and Secondary Schools (1984), provided educators with 
five themes central to the teaching and learning of 
geography.* This document was disseminated broadly to 
teachers and geography educators. 

Also in the mid-1980s, the National Geographic 
Society, through commentaries in the National 
Geographic magazine, began calling for substantial 
changes in American attitudes toward geography and 
the United States educational system.* In addition, the 
National Geographic Society and the National Center for 
Education Statistics (NCES) commissioned a NAEP 
geography assessment of high school seniors in 1988.* 
The information provided by this assessment, along w'.lh 
data from a 1988 Gallup survey of geographic literacy, 
suggested the need for significant reform of geographic 
education in the United States. Both surveys indicated 
that student knowledge and skills were tar short of what 
was needed for responsible and productive citizenship.® 

i % 

When President Bush convened the nation’s governors 
at the education summit in 1989, the value of geography 
and the need for better geography education were further 
emphasized when geography was established as one of 
five core subjects in the nation’s schools. Subsequently 
NAEP was authorized to conduct a comprehensive 
assessment of geography in 1994 at grades 4, 8, and 12. 

Overview of the 1994 National 
Assessment of Educational l^ogress 
" (NAEP)'" ' ■: 

A project of the NCES, NAEP collects valuable 
formation about what students know and can do in 
! difrerent curricular areas. Since being initiated by 
Congress in 1969, NAEP has carried out its federally 
supported mandate as the only ongoing national 
assessment of student achievement. Both public and 
nonpublic school students in grades 4, 8, and 12 are 
regularly sampled and assessed in reading, history, 
geography, mathematics, science, writing, imd other 
subjects. The assessments are based on content 
frameworks developed through a national consensus 
process involving teachers, curriculum experts, parents, 
and members of the general public. The frameworks 
attempt to maintain a balance between current 
instructional efforts, curriculum reform, research 
results, and desirable levels of achievement 








The NAEP 1994 geography assessment was admin- 
istered to national samples of fourth-, eighth-, and 
twelfth-grade students attending public and nonpublic 
schools. In all, approximately 19,000 students were 
assessed. Students’ geography performance is described 
on a scale ranging from 0 to 500 and in relation to three 
levels of achievement: Basic, Proficient, esti Advanced. 
The assessment results are reported for national popula- 
tions and for specihc subgroups. 



The NAEP 1994 Geography 
Framework 

The structure and content of the assessment were 
determined by the Geography Framework for the 1994 
National Assessment of Educational Progress.^ Although 
geography had been assessed at the grade 12 level in 
1988, the 1994 framework offered a new description of 
the content of the discipline and mandated a new format 
for the assessment This forward-looking framework was 
developed under the auspices of the National Assessment 
Governing Board (NAGB) through a consensus process 
managed by the Cotmcil of Chief State School Officers. 
The consensus process involved more than 50 educa- 



tors, policymakers, professional geographers, represen- 
tatives of the business community, assessment experts, 
and curriculum specialists. In addition, several hundred 
other experts and interested members of the public 
contributed to the development process, either by 
participating in hearings or by reviewing drafts of 
documents. 

The NAEP 1994 geography framework is organized 
along two dimensions, a content dimension and a ' ' ‘ 
cognitive dimension. The three content areas of the 
framework — Space and Place, Environment and 
Society, and Spatial Dynamics and Connections — 
served to clarify specifics of subject matter that were 
measured in the 1994 geography assessment. 

The cognitive dimension of the framework specified 
areas of thinking expected of students as they embrace 
specific geography content These cognitive areas were 
defined as Knowing, Understanding, and Applying. 

As Figure 1.1 illustrates, the content and cognitive 
dimensions of the framework form a matrix: The 
assessment addresses each cognitive process in each 
content area. The content and cognitive areas of the ' 
geography framework are described in greater detail in 
the following sections. .. ' ■ 
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Geography Content Areas 

T§ree geography content areas form the core organizing 
structure of the framework* The content areas were 
intended to ensure that all branches of geographic study 
were covered and that emphases on various areas were 
balanced. The content areas are also used to define 
subscales for reporting. The content areas, as described 
in the assessment framework, are summarized below.’' 

1. Space and Place: Knowledge of geography as it 
relates to particular places on Earth, to spatial 
patterns on Earth’s surface, and to physical and 
human processes that shape such spatial patterns* 
The study of space and place is basic to geography. 
Space is the basic resource and organizing element for 
the discipline. Patterns that are illustrated on maps 
reflect both natural features and human activities. This 
content area requires students to distinguish between 
and understand the spatial distribution of physical and 
human characteristics. To accomplish this, they must 
locate significant features and places on Earth, recognize 
existing patterns in the distribution of features and 
places, and comprehend the reasons for the development 
and existence of these patterns. 

2. Environment and Society: Knowledge of geography 
as it relates to tiie interactions between environment 
and society. 

Geography is an integrative discipline that focuses on 
the interrelationships between the physical environment 
and society. Human adaptation to and modifiwtion of 
the environment have b^n of continual and incre^ing 
importance to economies and polities. Underst^ding 
the nature, scale, and ramifications of such environ- 
mental transformations is fundamental in geography 
education, and is the core of this content area. 

‘ Students must, be aware that every environmental 
issue lends itself to many interpretations, depending on 
people’s perspectives. Students must consider such 
multiple perspectives as they evaluate decisions about 
issues such as land use and resource development, 
because the results of such decisions often luve compli- 
cated and unpredictable consequences. Making wise 
decisions concerning the costs and benefits of such 
environmental modification is an expressed goal of 
geography education. 



Finally, students must understand the causw and 
effects of natural hazards and disasters on the livability^ 
of certain areas and that the phrase “a safe place to live” 
is subject to personal interpretation. 

3. Spatial Dynamics and Connections: Knowledge of 
geography as it relates to spatial connections 
among people, places, and regions. 

This content area requires students to demonstrate 
comprehension of cultural, economic, and political 
regions and the connections among them, and explores 
critical problems in human interaction. Students must 
understand how peoples and places are alike and how 
they differ. They must demonstrate that a coniprehen- 
sive understanding of these similarities and differences 
can contribute to our performance as citizens on both a 
personal and broad international scale. 

Students should know that people of every nation 
are increasingly connected to and dependent upon other 
peoples and places of the world for both human and 
natural resources. In this content area, students must 
demonstrate the knowledge that the world’s resources 
are unevenly distributed, and an understanding of 
how this contributes to the movements of people, 
patterns of trade, and patterns of conflict Students also 
should understand the increasing significance of human 
interdependence as various populations are searching 
for clearer identities and independence. 

Tbble 1.1 shows the percentages of ass^ment time 
to be devoted to each content area, as speafied in the 
framework. • 



DistrfiHitiMof AssessmotlintAcfoss 
Geogrophy Cwttwt ' ' 


CwtMt Atm 




SpeaondFlm 




EnvimnmMtandSodity 


30X 


Spald Dyixxnks and CwHWcKons 


• 3W 



dtSTGUPYAVAIUBLE 



n 19 






Geography Cognitive Areas 

ThfrNAEP geography framework also outlines three 
cognitive areas to be assessed: Knowing, Understanding, 
and Applying. These are described below.^ 

1. Knowing — What is it? Where is it? 

In this area, students are assessed on their ability 
to perform two related functions concerning information: 
(1) an observation function and (2) a recall function. 

In general, tasks in this cognitive area are meant to 
measure students’ ability to observe different elements 
of the landscape and to answer questions by recalling, 
for example, the name of a place or a resource 
indigenous to a particular country or by finding 
information about trading patterns among several 
countries. 

2. Understanding — Why is it there? How did it get 
there? What is its sig^cance? 

In this area, students are asked to attribute meaning 
to what has been observed and to explain events. Explain- 
ing events and placing them in context requires 
students to demonstrate the ability to comprehend, to 
see connections among diverse bits of geographic 
information, and to use that information to explain 
existing patterns and processes on Earth. 

3. Applying — How can knowledge and understanding 
be used to solve geognq>hic problenu? 

In this area,' students were involved in a range of 
higher order thinly sldlls. Students are asked to 
classify, hypothesL^ use inductive and deductive 
reasoning, and form problem-solving inodels. This area 
of thinking calls on, students to. use many tools and skills 
of geognq>hy as they attempt to develop a comprehen- 
sive understanding of a problem en route to proposing 

viable solutions. ..s;. . , , 

^ * 

The cognitive areas are not used as reporting 
subscales, but rather were used in the assessment 
construction process to ensure appropriate coverage. 
Ihble 1.2 shows the percentages of the assessment time 
to be devoted to each cognitive level, as specified in the 
framework. 
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The NAEP 1994 Geography 
Assessment Instniment 

Guided by the NAEP 1994 geography framework, test 
development staff at Educational Testing Service worked 
to craft an assessment that measured the range and 
breadth of skills central to geography. As a national 
assessment, it was imperative that the NAEP 1994 
geography assessment reflect the scope of perspectives 
and opinions among educators about geogr^hy content 
and how it should be assessed. Therefore, the assessment 
development process encompassed an extensive series of 
reviews conducted by content and measurement experts, 
teachers, and researchers. All components of the 
assessment were evaluated for curricular relevance, 
developmental appropriateness, and adherence to the 
framework and test specifications. 

The resulting assessment had many innovative and 
interesting features, each designed to allow for accurate 
measurement of the domain. Some of the most notable 
of these features are described below. 



► 



► 




As called for in the framework, over half of student 
assessment time was devoted to constructed- ~ 
response questions. Some of these questions 
required short responses (one or two sentences), 
while many required more extended answers (a 
paragraph or more). ff 

The ass^ment used a wide range of stimulus ’ ' 

materials designed to assess spatial and interpretiw 
skills intrinsic to geognq)hy and to engage students ^ 
as actively as possible in the assessment Stimuli 
included maps, charts, cartoons, tables, text - 
historical documents, and photographs. ’ * 



One sec^on of the assessment at each grade was , 
designed to assess students’ ability to work with 
atlases. 



I ’.'1 



■ ■ ' It* ' ’ * , - r r ’ r I '■ , - 

^ Constructed-response questions were not limited to 
those that required students to produce written _ ' 
responses. At each grade, a certain number of ' 
questions asked students to create their own inaps, 
charts, or other data displays. 



^ In addition to portions of the assessment that were 
structured as broad surveys of knowledge and skills, 
certain sections were designed to allow students to 
work in-depth on a particular topical are4. 
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In addition to multiple-choice questions, the 
assessment included both short and extended 
cqnstructed-response questions. The percentage of 
response time devoted to answering constructed- 
response questions was approximately 60 percent at 
each grade. Each constructed-response question was 
scored according to a scoring guide, or rubric, that gave 
credit for partially correct answers. The exercises called 
for a range of responses. Short constructed-response 
questions called for a word, a phrase, or a sentence or 
two to demonstrate understanding of specific material. 
Extended constructed-response questions called for 
more developed argument, gathering of evidence, or 
interpretation of data. Short constructed-response 
questions were scored according to three-part scoring 
guides, in which a score of 3 represented an appropriate 
answer, a score of 2 a partially correct answer, and a 
score of 1 an injq)propriate answer. Extended 
constructed-response questions were scored according 
to four-part scoring guides, in which a 4 was assigned to 
complete responses, a 3 to responses that responded to 
essential components of the task, a 2 to partially correct 
responses, and a 1 to inappropriate answers. Many of Ae 
constructed-response questions were scaffolded; that is, 
students were asked to respond first to one portion of the 
question and then to another. In a few cases, students 
were asked to do other tasks, such as drawing a map. 

A national field test of assessment exercises was 
conducted prior to use in the assessment to ensure their 
appropriateness for asse^in^ geography knowledge. 
Statistical analyses and qualitative reviews were 
conducted in selecting materials from the field test to be 
included in the assessment The NAEP 1994 geography 
assessment that emerged firom this broad consensus and 

development effort repre«nted a new generation of 
large-scale geography asis^ments. 

Across the three grades assessed — fourth, eighth, 
and twelfth — a total of 228 multiple-choice, 80 short 
constructed-response, and 30 extended constructed- 
response questions comprised the NAEP1994 geography 
assessment For efficiency, some questions were 
administered at two grade levels. The assessment design 
is discussed in more detail in Appendix A and in the 
NAEP 1994 Ikchnical Report. 



Description of School and Student 
Samples 

As with all NAEP assessments, the schools and students 
participating in the 1994 geography assessment were 
selected through scientifically designed, stratified 
random sampling procedures. Approximately 19,000 
students in 1,500 public and nonpublic schools across 
the country participated in the geography assessment 

The results presented in this report are based on 
representative samples of students at each of the three 
grades. Each selected school that participated in the 
assessment and each student assessed, represented a 
portion of the population of interest As a result the 
findings provided in this report pertained to all fourth, 
eighth, and twelfth graders in the nation. (For a more 
detailed description of the sample and the sampling 
procedures, see Appendix A.) 

Reporting NAEP Geography Results 

The NAEP 1994 geography assessment provides a wealth 
of information on the geography abilities of the nation’s 
fourth-, eighth-, and twelfth-grade students. To 
maximize usefulness to policymakers, educators, 
parents, and other interested parties, the NAEP results 
are presented both as average scores on Ae NAEP 
geography scale, and in terms of the percen^e of 
students attaining NAEP geography achievement levels. 
Thus, NAEP results not only provide infomiation abotrt 
what students know and am do, but alw indicate 
whether their achievement meets' eq^fectations of ‘ ’ 
what students should know and should^ able to do. ' 
Furthermore, the descriptions of skills and abilities 
oq^ected of students at each achievement 1(^1 , ‘ ^ ; 

help make the reporting of assessment results ' 
more meaningful. ; . 
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The Geography Scale 

Student responses to the NAEP 1994 geography assess- 
ment were analyzed to determine the percentage of 
students responding correctly to each multiple-choice 
question and the percentages of students responding in 
each of the score categories for constructed-response 
questions. Item response theory (IRT) methods were 
used to produce within-grade scales that summarize 
results for each of the three geography content areas 
described earlier. Each subscale for grade 4 was linked 
to the corresponding subscale for grade 8. Likewise, 
each subscale for grade 12 was linked to the correspond- 
ing subscale for grade 8. Then, each linked subscale was 
mapped onto a scale with a range of 0 to 500. These 
separate subscales were then weighted by the percent- 
ages shown in Tbble 1.1 to produce a composite NAEP 
geography scale. Chapters 2, 4, and 5 present results 
based on the geography scale. (The scales for each of the 
NAEP subjects assessed in 1994 — reading, U.S. history, 
and geography — were developed independently. 
Therefore, results should not be compared across sub- 
jects. Details of the scaling procedures are presented in the 
NAEP 1994 Technical Report.) 

' Geography Achievement Levels 

The 1994 assessment results are also reported using the 
geography achievement levels as authorized by the 
NAEP legislation and adopted by the NAGB. The 
achievement levels are based on collective judgments ; 
gathered about what students should know and should 
be able to do relative to the body of content reflected in 
the NAEP assessment firework. Three achievement , , 

■ levels were defined for each grade level assessed: Basic, 
Proficient, dxAAdvanced^lh& levels were defined by a 
I broadly representative panel of teachers, education 

specialists, and members of the general public. 

For reporting purposes, the achievement levels for 
each grade are placed on the NAEP geography scale 
defining in four ranges — Basic, Proficient, Advanced, 
and the region below Basic. Figure 1.2 presents the 
policy definitions of the three achievement levels. 
Chapter 3 contains specific descriptions for the geogra- 
phy achievement. 

I It should be noted that the setting of achievement 

j levels on the national assessment is relatively new and 
in transition. Some evaluations have concluded that the 
i percentage of students at certain levels may be 

underestimated.* On the other hand, critiques of those 
evaluations have found that such conclusions are not 
supported by the weight of the empirical evidence.® 
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Basic This level denotes partial mastery of 

prerequisite knowledge and skills diat are 
iimdantental for proficient work at each 
grade. 

Proficient This level represents solid academic perfor- 
mance for each grade assessed. Students 
' ' reaching this level have demonstrated 

competency over challenging subject matter, 
including subject-matter knowledge, 
<q)plication of such knowledge to real-worid 
' - situations, and analytical skills appropriate 
to the subject matter. 

Advanced This level signifies superior performance. 



The student achievement levels in this report have 
been developed carefully and responsibly, and have been 
subject to refinements and revisions in procedures as 
new technologies have become available. Upon review of 
the available information, the Commissioner of NCES 
has judged that the achievement levels are in a 
developmental status. However, the commissioner and 
the NAGB also believe that the achievement levels are 
useful and valuable in reporting on the educational 
achievement of students in the United Stat^. Results 



reported in terms of the geography achievement levels 
are presented in Chapter 3 of this report. ’ j; : : . ; 



Interpreting NAEP Resuits^ift 






The average geography scale scores and the percentages 
presented in this report are estimates beatjitt jdiey f 
are based on samples rather than on the 
population(s). As such, the results are siib}^ to a ^ 
measure of uncertainty, reflected in the s^dard errors 
of the estimates. These standard errors are pre^nted in 
parentheses along with the estimated average scale 
scores or percentages in tables throughout this report 



The signihcant differences discussed in toe 
following chapters take into account the stah^d errors 
associated wito the estimates. The compansons are 
based on statistical tests that consider both the 
magnitude of the differences between the average scale 
scores or percentages and the standard errors of those 
statistics. Throughout this report, differences are 
defined as significant when they are significant from a 
statistical perspective. This means that observed 
differences are unlikely to be due to chance factors 
associated with sampling variability. All differences 
reported are significant at the .05 level with appropriate 




adjustments for multiple comparisons. The term 
“significant,” therefore, is not necessarily intended to 
imply judgment about the absolute magnitude or 
educational relevance of the differences. The term is 
intended to identify statistically dependable population 
differences as an aid in focusing subsequent dialogue 
among policymakers, educators, and the public. 

The use of a common, cross-grade me tric for the 
three geography subscales and the composite geography 
scale was primarily for ease of reporting. However, the 
methodology used to produce the 0 to 500 scales used to 
report fourth-, eighth-, and twelfth-grade results 
(described in ^pendix A) may not allow meaningful 
comparisons across grades. Similarly, scale score 
differences (e.g., between subscale or composite scale 
aver^es for males an j females) should probably not be 
compared across grade. The reader is best served by 
focusing on within-grade group comparisons and 
inferences. 

Cautions in Interpretations. The reader is cautioned 
against using the NAEP results reported herein to make 
simple or casual inferences related to subgroup 
membership or the effectiveness of public and nonpublic 
schools. For example, performance differences observed 
among racial/ethnic subgroups are almost certainly 
associated with a broad range of socioeconomic and 
educational factors not discussed in this report and 
possibly not addressed by the NAEP assessment program. 
Similarly, differences between public and nonpublic 
schools may be better understood after accounting for 
factors such as composition of th student body, 
parents* highest level of education, and parental 
interest 

Sample Questions from the 
NAEP 1994 Geography Assessment 

As discussed earlier, the NAEP 1994 geography 
assessment is a rich collection of exercises developed to 
survey the geographic knowledge and skills of students 
in grades 4, 8, and 12. Each student received a mixture 
of multiple-choice and constructed-response questions. 

Figure 1.3 presents samples of assessment exercises. 
(Additional example questions are shown in Appendix 
C.) The tables accompanying the exercises present two 
types of percentages: (1) the overall percentage of 
students within a grade who answered the question 
correctly and (2) the percentages of students within 
each of the achievement level intervals — Basic, 
ProBcient, dx\A Advanced as well as the percentage of 



students below Basic — who answered the questions 
successfully. (A fuller description of the achievement 
levels can be found in Chapter 3. At grades 4 and 12, 
the percentages for students at the Advanced level are 
not presented for these questions because of small 
sample sizes.) 

The first exercise presented in Figure 1.3 is a short 
constructed-response question administered at grade 4 
that asks students to list an advantage and disadvantage 
of a particular method of waste disposal. A sample 
response for a student who received a score of 
“Complete” (defined as a score of 3 on a three-part 
scoring rubric) on this question is also provided. The 
table shows that few fourth graders (11 percent) 
provided answers that received a score of “Complete.” 

Less than 1 percent of those students below Basic and 
only 8 percent of those who scored within the Basic 
achievement level interval provided answers rated as 
“Complete.” Almost one-quarter (28 percent) of fourth 
graders who scored within the Proficient achievement 
level interval provided answers rated as “Complete.” 

The second exercise presented in Figure 1.3 is a 
multiple-choice question administered at grade 8. The 
table shows that about one-fifth (21 percent) of all 
eighth graders answered this question correctly. 
Seventeen percent of eighth graders who scored below 
the Basic achievement level answered the question 
correctly, compared to 19 percent for students who 
scored within the Basic level. One-quarter of students 
who scored within the Proficient achievement level 
interval and nearly half (47 percent) of students who 
scored within the Advanced level answered the question 
correctly. 

The third exercise shown in Figure 13 is an 
extended constructed-response question administered at 
. grade 12. The questions ask students to select a site for a 
proposed shopping center and to support their selection 
with available information. (Neither site alternative was 
“correct”; student responses were scored based on the 
students’ ability to support their selection.) Sample 
responses for students who received scores of 
“Essential” and “Complete” (scores of 4 or 3 on a four- 
part scoring guide) are provided. For this extended 
constructed-response question, over half (55 percent) of 
all twelfth graders provided answers rated “Essential” or 
better. For those twelfth graders who scored below the 
Basic achievement level, 27 percent provided a:\iwers 
rated “Essential” or better, compared with 57 perc nt of 
students within the Basic level. Almost three-qviarters 
(76 percent) of twelfth graders who scored within the 
Proficient achievement level interval provided answers 
rated “Essential” or better. 
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Sample Response (Scote of 3) 

WAYS TO GET RID OF WASTE 

. Dumping far out in the ocean 

• Burning 

• Recycling 

• Burying in landfills 

From the list above, select one method of getting rid of w®ste and 
identify one advantage and one disadvantage of this method. 

Method of waste disposal: . 

Advantage: ^ 



Djaadvamagr nnttj j 

.^kj Ca^ /mnlSiy iL/ 



C/rAdu 



A Complete response (score of 3) accurately describes an advantage and disadvantage of one 
method of waste disposal. Explanations should be both specific to that method and geographically 
logical. 
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Example of a Grade 8 Multiple-Choice Question (> indicates correct answer) 

In the United States, most of the fertile soils of the Midwest were derived from 

►A glaciers 
B volcanic activity 
C decaying organic matter 
D eroded sandstone 

Geography Content Area: Environment <md Society 
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Example of a Grade 12 Extended Con*tructed-Re$pon$e Question 
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MADDIEVOXE 




MaddievUle is building a new shopping center. There is a disagreement in the 
city council over whether to build the shopping center at site A or at site B on 

.themap. . ..ir ■■■ ■ 

As a resident of the city who would like to shop at the new shopping center, 
write a letter to the miomr in support of eitiier site A or site B. Give three 

reasons why the site you support is better than the other site. 



Geography Content Area: Spatial Dynamics and Connections 
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Sample Response (Score of 3) 

Maddieville is building a new shopping center. There is a disagreement in the 
dty council over whether to build the shopping center at site A or at site B on 
the map. 

As a resident of the dfy who would like to shop at the new shopping center, 
write a letter to the mayor in support of either site A or site B. Give three 
reasons why the site you support is better than the other site. 
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An Essential response (score of 3) chooses a site and supports the choice with two reasons. 
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Sample Response (Score of 4) 

Maddieville is building a new shopping center. There is a disagreement in the 
dfy council over whether to huUd the shopping center at site A or at site B on 
the map. 

As a resident of the dfy who would like to shop at tiie new shopping center, 
write a letter to the mayor In support of either site A or site B. Give three 
reasons why the site you support is better than the other site. 
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A Complete response (score of 4) chooses a site and gives three reasons for the choice. 
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Geography Results for the 
Nati in and Regions 

Introduction 

This chapter presents the overall NAEP geography 
scale scores of students in grades 4, 8, and 12. 

Findings are presented for the nation, regions of the 
country, and selected subgroups of students. Student 
performance is reported on the NAEP geography scale, 
j v.'hich ranges from 0 to 500. (For a visual representation 
of student performance on each of the geography 
j content subscales — also ranging from 0 to 500 — 

. see Chapter 5.) 

I The results provided in this chapter address 

I statistically significant differences that were found 

between reporting subgroups. In other cases, score 
estimates for various subgroups may appear to differ, 
but these differences are not statistically signihcant. 
(Significant differences are those that are unlikely to 
be due to sampling variability or chance.) All significant 
differences among reported subgroups are indicated to 
provide a comprehensive and balanced discussion of the 
results. However, when reading this report, statistical 



significance should not necessarily be equated with 
educational or instructional significance. 

In addition, the NAEP 1994 geography assessment 
data are explored in more depth by examining the 
interactions among several major reporting variables. 
Average geography scale scores are examined for 
subgroups of students within various demographic 
populations. By doing so, it is possible to determine 
whether general patterns of geography performance for 
certain groups of students are related to additional 
background characteristics. 



Geography Results for the Nation 

T^le 2.1 presents the average geography scale scores for 
fourth-, eighth-, and twelfth-grade students across the 
nation attending both public and nonpublic schools. 

^ At grade 4, the average geography scale score was 
206. The bottom 10 percent of the population scored 
at or below 146 and the top 10 percent scored at or 
above 257. 

^ At grade 8, the average score was 260 . The bottom 
10 percent of the population scored at or below 213 
and the top 10 percent of the population scored at or 
above 302. 

► At grade 12, the average geography scale score was 
285. The bottom 10 percent of the population scored 
at or below 244 and the top 10 percent scored at or 
above 321. 
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l^Ie 2.2 and Figure 2.1 present results for the 
nation as well as the four regions of the country: 
Northeast, Southeast, Central, and West (The 
composition of the regions is described in Appendix A.) 
At grade 4, students in the Central region outperformed 
those in the other three regions. At grade 8, the 
students in the Northeast and Central regions had higher 
average scores than their counterparts in the Southeast 
and West Among grade 12 students, those in the 
Southeast had lower average scores than did those in 
each of the other regions. 



Geography Results for Major 
Reporting Subgroups 

Tables 2.3 through 2.8 present the average geography 
scale score estimates for major subgroups of the fourth-, 
eighth-, and twelfth-grade student populations. 

Cautions in Interpretation. In interpreting the results 
presented in this section, the reader is cautioned against 
making simple or causal inferences related to subgroup 
membership or effectiveness of Title I programs or 
public and nonpublic schools. Performance differences 
among groups of students may result from differences 
in socioeconomic status and home background variables. 
For example, differences observed among racial/ethnic 
subgroups can almost certainly be associated with a 
broad range of socioeconomic and educatioi^ factors. " ’ 
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Figwrt 2.1 Average NAEP Geogro|iby Scale Sceres by Grade and by Regtoa 
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Similarly, differences between public and nonpublic 
schools may be better understood if differences in 
composition of the student body, parents’ highest level 
of education, and parental involvement in education are 
considered. 

Race/Ethnicity. As part of the background questionnaire 
that was administered in conjunction with the NAEP 
1994 geography assessment, students were asked to 
identify themselves as belonging to one of six mutually 
exclusive categories: White, Black, Hispanic, Asian, 
Pacific Islander, and American Indian (including Alaskan 
Natives). Thble 2.3 presents the average scale scores for 
racial/ethnic subgroups. The geography assessment, 
like other NAEP surveys, revealed substantial variation 
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in performance among the different racial/ethnic 
subgroups. At all three grades, the average scores of 
White and Asian students were significantly higher than 
those of Black and Hispanic students. In addition, at all 
three grades, the average scores of Hispanic students 
were higher than those of Black students. 

At grade 4, White and Asian students also 
outperformed American Indian students. American 
Indian students exhibited a higher average score than 
Black students. Finally, at grade 4, Pacific Islander 
students scored higher, on average, than did Black and 
Hispanic students. 

At grades 8 and 12, the samples do not permit 
accurate determination of the standard errors associated 
with the average scale scores of Pacific Islander and 
American Indian students. For this reason, differences 
between these and other population subgroups are not 
discussed. 

Gender. Thble 2.4 presents the average geography scores 
for male and female students in grades 4, 8, and 12. At 
all three grades, male students had sigiuficantly higher 
average scale scores than did female students. Specifically, 
the average score for male students was five points 
higher than female students at grade 4, four points 
higher at grade 8, and seven points hi^er at grade 12. 
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Parents* Highest Level of Education* Students were 
asked to report on the education level of their parents or 
guardians: did not hnish high school, graduated 
firofh high school, has some education after high 
school, or graduated from college. (Those who did 
not have this information chose the response option 
“I don’t know.”) The level of parents’ education 
discussed in this section is the highest level reported 
by students for either parent 

Before reviewing the findings for parental 
education, it should be noted that approximately one in 
three fourth graders and one in ten eighth graders 
reported not knowing the highest level of education 
attained by either of their parents. Furthermore, 
research has questioned the accuracy of student 
reported data among similar groups of students.^ 

These caveats notwithstanding, the NAEP geography 
results indicate a strong positive relationship between 
parental education and student achievement The results 
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(summarized in Thble 2.5) show that as a general rule, 
groups of students reporting given levels of parental 
education had higher average scores than all groups 
reporting lower levels of education. The only exception 
to this pattern occurred at grade 4. There was no 
significant difference in performance between fourth 
graders who reported that a parent had completed 
college and those who reported that a parent had 
completed some education after high school. 

J^pe of Location. Each participating school in the 
NAEP 1994 geography assessment was classihed 
according to its geographic location. The three types of 
location — central city, urban fringe/large town, and 
rural/small town — are based on U.S. Bureau of the 
Census definitions of sttmdard metropolitan statistical 
areas, population size, and density. These categories 
indicate the geographic location of the students’ school 
and are not intended to indicate or imply social or 
economic meanings for these location types. (The type 
of location classifications are described in Appendix A.) 
Table 2.6 presents results for students attending schools 
in each type of location. 
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The relationship between this variable and 
performance differs somewhat across the three grades. 
Aggrades 4 and 8, students attending schools in central 
city locations had significantly lower average scores 
than did students in other types of locations. At grade 
12, students attending schools in urban fringe/large 
town locations performed at a significantly higher level 
than those attending schools in other locations. 

Title / Participation, Staff members at each school that 
took part in the NAEP 1994 geography a^essment were 
asked to identify which of the students participated in Title I 
programs or received services funded by Title I grants.^ 
The Title I legislation provides funds to state and local 
educational agencies to support mathematics and 
reading programs and initiatives aimed at assisting 
disadvantaged students (those who are failing or are at 
risk of failing) in low-income communities. 

Thble 2.7 presents the results for students who 
received Title I services and for those who did not. As 
stated earlier, differences in performance between these 
recipients and nonrecipients should not be viewed as an 
evaluation of Title I programs. Typically, Title I services 
are intended for students who score poorly on 
assessments. 
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The percentage of students receiving Title I services 
was larger at grade 4 (13 percent) than at grade 8 
(7 percent). At grades 4 and 8, the average geography score 
W recipients of Title I programs was significantly lower 
than the average score for nonrecipients. (Differences 
for twelfth graders are not discussed here because the 
nature of the grade 12 sample does not allow for 
accurate estimation of the variability of the percentage 
and average scale score for Title I recipients.) 

Thpe of School, The national assessment collects data 
on students attending both public and nonpublic 
schools. Nonpublic schools include Catholic, other 
religious, and private institutions. As shown in Thble 
2.8, students in nonpublic schools outperformed 
students in public schools at all three grades. 

As noted in Chapter 1, the reader is cautioned 
against using these results to make simplistic inferences 
about the relative effectiveness of public and nonpublic 
schools. Average performance differences between the 
two types of schools may, in part, be related to socioeco- 
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nomic and sociological factors, such as levels of parental 
involvement in their child’s education. To get a clearer 
picture of the differences between public and nonpublic 
schools, more in-depth analyses must be undertaken. 



An In-Depth Look at Selected 
Background Characteristics 

One way to take a closer look at the performance of 
students within selected demographic populations is to see 
if the magnitude of the differences among groups of 
students varies when other background characteristics 
are taken into account. This section presents NAEP 
geography results for subgroups of students within 
various demographic populations. Three specific 
background characteristics are explored with these 
analyses: gender, race/ethnicity, and parents’ highest 
level of education. In addition, type of school (public 
and nonpublic) results conditioned on parents’ 
education are examined. 



Many factors can influence differences among 
subgroups. While looking at some of the NAEP variables 
concurrently yields interesting results, these results can 
be due to a variety of circumstances that cannot be 
controlled for in a large-scale assessment such as NAEP. 

Gender and RacelEtiinicitg, The first two student 
characteristics examined are gender and race/ethnicity. 
As mentioned earlier, at all three grades, male students 
outperformed female students, and White students 
displayed higher average scores than Black or Hispanic 
students. (Asian, Pacific Islander, and American Indian 
students are not included in this section because, for 
the purposes of this analysis, their sample sizes are 
insufficient) One question that might be asked is 
whether or not this pattern in male and female students’ 
scores holds regardless of race/ethnicity. Furthermore, 
was the difference in performance between male and 
female students larger in some racial/ethnic subgroups 
than in others? 
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T^le 2.9 presents results of analyses carried out to 
answer these questions. Average scores of male and 
f^ale students and the differences between these 
averages are presented separately for three racial/ethnic 
subgroups. Overall, male students outperformed female 
students across all three grades. However, when the 
racial/ethnic subgroups are examined, the differences 
for White students at grades 4 and 8, and all three racial/ 
ethnic groups at grade 12 were significant. 

A comparison of the magnitude of the average score 
differences between male and female students yielded 
no evidence that these differences varied significantly 
across racial/ethnic groups of students. 

Gender and Parental Education, Thble 2.10 presents an 
analysis of gender differences in geography performance 
relating to parents' highest level of education. Average 
scores of male and female students and the differences 



between these averages are presented separately for the 
different levels of parental education reported by 
students. One question that can be answered with these 
data is whether or not differences in average geography 
scores between male and female students were evident 
for students at all levels of parental education. 
Furthermore, were gender differences larger at some 
parental education levels than at others? 

Consistent with the overall gender results, average 
scale scores at grade 12 were higher for males than for 
females across all levels of parental education. Also, at 
grade 4 for students reporting that at least one parent 
graduated college, males outperformed females. 

Again, a comparison of the magnitude of the average 
score differences between male and female students 
yielded no evidence that these differences varied signifi- 
cantly across parental education levels. 
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1^ of School (Public and Nonpublic I and Parental 
Education, As reported earlier in this chapter, at all 
grades students attending nonpublic schools had signih- 
canlly higher average scores than those attending public 
schools. Was this difference in geography performance 
between nonpublic and public schools evident at all levels 
of parental education? Furthermore, was the difference 
in performance larger for some levels of parental 
education than for others? 

Ihble 2.11 presents results of analyses carried out to 
address these questions. One striking result presented in 
Ihble 2.11 is that the sample sizes for students attending 
nonpublic schools whose parents did not graduate high 
school were too small to estimate an average score. 
Therefore, discussions of public/nonpublic differences at 
this level of parental education are not possible. As with 
the overall results, students attending nonpublic schools 
in each grade had higher average scores than their 
public school counterparts across all levels of parental 
education. All of these differences were significant, with 
the exception of twelfth graders who reported that at 
least one parent had some education after high school. 



(Again, the reader is cautioned against drawing causal 
inferences from these results.) 

A comparison of the magnitude of the average 
score differences between nonpublic and public school 
students yielded little or no evidence that these 
differences varied significantly across levels of 
parental education. 

RacetEthnicity and Parental Education, Racial/ethnic 
differences in twelfth-grade average geography scores 
are presented in relation to parental education levels in 
Table 2.12. The average scores of White, Black, and 
Hispanic students and the differences between those 
scores are presented separately for the different levels of 
pcurents’ education reported by students. One question 
that can be answered with' these data is whether or not 
the differences in average geography scores among 
White, Black, and Hispanic students were evident for 
students at all levels of parental education. In addition, 
were the performance differences between racial/ethnic 
groups larger at some parental education levels than at 
others? 
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At the twelfth grade, the average scores of White 
students was sign'f.canliy higher than that of Black or 
Hispanic students across all levels of parental education. 
In addition, Hispanic twelfth graders significantly 
outperformed Black students at all levels of parental 
education. (Data for fourth and eighth graders are not 
presented in this tabulation because of wide variation 
among racial/ethnic groups in the accuracy of reporting 
parental education. For further details, see discussion in 
Appendix A under “Parents’ Education Level.”) 

A comparison of the magnitude of the score 
differences among White, Black, and Hispanic students 
yielded no evidence that these differences varied 
significantly across levels of parental education, with 
one exception. The magnitude of the difference in 
geography scores between White and Black twelfth 
graders whose parents had some education after high 
school was less than the magnitude of the difference 
between White and Black students whose parents were 
college graduates. 

The data in Tbble 2.12 show that the racial and 
ethnic differences in average geography scores, as in 
the other subjects assessed by NAEP in 1994, persist 



across different levels of parental educational 
attainment. These NAEP patterns run somewhat 
counter to previous findings from other studies. The 
National Education Longitudinal Study of 1988, with 
more complete measures of socioeconomic status, found 
substantial reductions in achievement differences 
associated with racial/e.thnic group membership after 
accounting for family resources.^ In addition, the 
College Board haf found that racial differences on the 
Scholastic Aptitude Test are diminished somewhat when 
family income differences are taken into account* So, 
the NAEP findings should be interpreted carefully in 
relation to these other results. 

In interpreting these findings, it is important to 
understand that student achievement is a result of 
multiple factors including educational ejq^erience, 
resources from the home, and the larger social 
environment. These factors may also differ depending 
on the students’ racial/ethnic groups and thus 
contribute — along with parents’ educational level — to 
achievement differences. Such factors might contribute 
to reasonable explanations for why parents’ educational 
levels might be associated differently with student 
achievement for different racial/ethnic groups. 
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Summai^y 

The NAEP geography assessment showed patterns of 
performance among reporting subgroups, some of 
whom consistently performed better than others. These 
patterns of performance were also evident in inter- 
actions among those reporting variables. 

> Among the different regions of the nation, students’ 
scores varied at each grade level. Grade 4 students 
from the Central region, for example, outperformed 
students from the other three regions. 

> In general, the higher the level of parental educa- 
tion, the higher ti<e level of student perform<mce. 

^ White and Asian students at all three grades had 
higher scores than did Black and Hispanic students. 
Also, Hispanic students outperformed Black 
students at all three grades. 

> At all three grades, male students scored higher 
than female students. At grade 12, these significant 
differences persisted across racial/ethnic subgroups 
and across all levels of parental education. 

Students in nonpublic schools outperformed public 
school students. These differences were significant 
regardless of parental education levels. The only 
exception was at grade 12 for students reporting 
parents had some education after high school. 



Endnotes 

1. Looker, E. D. (1989). Accuracy of proxy reports of 
parental status characteristics. Sociology of 
Education, 62(4), 257-76. 

2. As a result of the Elementary and Secondary 
Education Act reauthorixed by Congress in 1994, the 
federal program formerly referred to as “Chapter 
One” was renamed “Title I.” 
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Geography Achievement Levels 

Introduction 

The average geography performance of our nation’s 

- students as presented in the previous chapter can be 

- explored further by considering the percentage of 
students who attained speciflc levels of achievement. 
Viewing students’ performance from this perspective 
provides some insight into the adequacy of students’ 
geographic knowledge and skills and the extent to which 
they are achieving expected levels of performance. 

The National Education Statistics Act of 1994 
requires that the National Assessment Governing Board 
develop “appropriate student performance levels” for 
reporting NAEP results. The NAEP law requires that 
these levels be “used on a developmental basis until the 
' Commissioner of Education Statistics determines . . . 
that such levels are reasonable, valid, and informative 
to the public.” It requires the Commissioner and 
the Board to make clear the developmental status of 
such levels. 
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The student achievement levels in this report have 
been developed and adopted by the National Assessment 
Governing Board, NAEP’s independent policy-making 
body, with contributions from a wide variety of 
educators, business and government leaders, and 
interested citizens. These levels of student achievement 
have been established to help Americans answer two 
questions that are important to parents and to ail 
citizens in the communities and nation in which we 
live. These questions are: “What should students know 
and be able to do as they progress and graduate from 
school?” and “How good is good enough in terms of 
student achievement on NAEP?” Answering these 
questions obviously involves judgments. The National 
Assessment Governing Board is not suggesting that 
there is a single answer to these questions. Rather, the 
Board is trying to put forward reasonable judgments 
that can inform citizens across America — information 
they can use to answer these questions in their own 
schools and communities. 



Developing carefully considered judgments about 
“what students should know and should be able to do” 
and “how good is good enough” is both difficult and 
. controversial. The Governing Board believes that these 
questions are so important that answers must be sought 
in an informed, responsible way. The process is subject 
to revision and refinement, as appropriate. 

The student achievement levels in ^ report approved 
by the Governing Board are the result of many hours of 
work. The levels are based on preliminary descriptions 
. . developed as part of the national consensus process to 
, determine the assessment design and content The 
Boitfd’s contractor, American College lasting (ACT), 

> which has extensive experience in standard-set^^ in 
many fields, designed the achievement l^f-setUng 
-^^pcess^This process was reviewed Iv ^Ires of individu- 
. rats, including policy-makers, professioi^^ 

^ 'teachers, parents, and other members ^tiie general 
‘ public. To develop the levels, ACT convimed a cross-’ ‘ 
section of educators and interested citizens from across 
the nation and asked them to recomnieiiH what students 
should know and be able to do in geography Prior to 
, , adopting these levels of student achievement, the Board 
engaged a large number of persons to comment on the 
recommended levels and to review the results. 



The result of the achievement level-setting pr<^(^ 

• is a set of achievement level descriptions^ a set of 
achievement level cutpoints on the 500-point NAEP 
scale, and exemplar questions. The cutpoints are scores 
that define the Basic, Proficient, an&Advmced 
achievement levels at grades 4, 8, and 12 on the NAEP 
scale. At present, evaluations conducted in other NAEP 
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subject areas on the level-setting process and critiques 
of these evaluations have provided mixed reviews. 
Theyefore, both the Governing Board and the 
Commissioner of Education Statistics regard the 
achievement levels as developmental; they should not be 
interpreted as statistically conclusive. Because these 
levels are still considered developmental, the reader of 
this report is advised to consider that status when 
interpreting the results. 1 he reader should recognize 
that the results are based on the judgments, approved by 
the Governing Board, of what advanced, proficient, and 
basic students should know ai d be able to do in each 
subject assessed, as well as on their judgments 
regarding what percent of students at the borderline for 
each level should answer each test question correctly. 
The latter information is used in translating the 
achievement level descriptions into outpoints on the 
NAEP scale. NCES uses these levels in reporting NAEP 
results, but it does not currently adjudicate the reliability 
or validity of these achievement levels. Rather they are 
reported directly as adopted by the Governing Board. 

The National Assessment Governing Board urges all 
who are concerned about “what students . hould know 
and be able to do” and “how good is good enough” to 
read and interpret these performance levels recognizing 



that this is a developing, judgmental process and is 
subject to various interpretations. The decision to include 
the levels in NAEP reports is an attempt to make the 
assessment results more useful for parents, educators, 
and policymakers by providing performance standards 
against which to measure educational progress. 

The three geography achievement levels — Basic, 
Proficient, and Advanced — were established by the 
National Assessment Governing Board for reporting 
NAEP results. As described in Chapter 1, the Basic level 
denotes partial mastery of prerequisite knowledge and 
skills that are fundamental for proficient work at each 
grade. The Proficient level represents solid academic 
performance and demonstrated competence over 
challenging subject matter. Ihe Advanced level signi- 
fies superior performance. The geographic knowledge 
and skills associated with each achievement level are 
described in Figure 3.1. 

The geography achievement levels attained by 
fourth-, eighth-, and twelfth-grade students in the NAEP 
assessment are presented in this chapter. Results are 
displayed for the nation, regions of the country, and 
major reporting subgroups. The cautions mentioned in 
Chapter 2 also are warranted when interpreting differences 
among subgroups. 



Rgor* 3.1 Gtograpliy AcUevMBMt Levels 

GRADE4 1 - ^ ~ ^ ~ 

Bask ' Students should be able to use words or diagrams to define basic geographic vocabulary; identify personal behaviors and 

(187) A perqrectives related to the environment and describe some environmenhil and cultural issues in their communis use visual 

' ' and technological tools to access information; identify major geographic features on maps and globes; be able to read and draw 

. r . ' -A ' simple maps, map keys and legend^ demonstrate how people depend upon, use, and adi^rt to the environment; arxl give 

■')f. .j examples ofthe movement ofpeople, goods, services, and ideas from one place to another. In addition to demonstrating an ' 

, understanding of how individuals are alike and different, drey should demonstrate a knowledge of the w:^ people depend on 

eachother. ■ . 

Proficient Students should be able to use fundamental geographic knowledge and vocabulary to identify basic geographic patterns and . .. ,v 

(240) , = processes; describe an environmental or cultural issue from more than one perspective; and read and interpret information ^ 

from visual and technological tools such as photographs, maps and globes, aerial photography, and satellite images* They 
. . should be able to use number and letter grids to plot specific locations; understand relative location terms; and sketch simple 

maps and describe and/or draw landscapes they have observed or studied. Proficient students should be able to illustrate how 
' people depend upon, adapt to, and modify the environment, describe and/or illustrate geographic aspects of a region using ' i 

fundamental geographic vocabulary and give reasons for current human migration; discuss the impact a location has upon 
cultural similarities and differences; and be able to demonstrate how an event in one location can have an impact upon 
another location. 

I Advanced Students should be able to use basic geographic knowledge and vocabulary to describe global patterns and processes; describe 

j (276) ways individuals can protect and enhance environmental quality; describe how modifications to the environment may have a 

I variety of consequences; explain differing perspectives that apply to local envirorunental or cultural issues; and demonstrate an 

^ understanding of forces that result in migration, changing demographics, and boundary changes. They should be able to solve 

i simple problems by applying information learned through working with visual and technological tools such as aerial and other 

photographs, maps and globes, atlases, news media, and computers. They should be able to construct models and sketch and 
label maps of their own state, the United States, and the world; use them to describe and compare differences, similarities, and 
patterns of change In landscapes; and be able to predict the impact a change in one location can have on another. They should 
be able to analyze the ways individuals and groups interact 
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Geography Achievement Levels 
^‘or *;he Nation and Regions 

T^ble 3.1 shows the percentages of fourth-, eighth-, and 
twelfth-grade students at or above the three geography 
achievement levels. At all three grades, roughly 70 percent 
of students scored at or above the Basic level. Roughly 
one-quarter of the students at each grade were able to 
reach the Proficient level. Few students at any grade were 
at the Acfe/ancecf level: 3 percent at the fourth grade, 

4 percent at the eighth grade, and 2 percent at the 
twelfth grade. (Note that the percentage of students below 
Basic is 100 percent minus the percentage at or above 
Basic.) 

There were several differences in regional performance 
on the geography assessment At grade 4, the percentage 
of students in the Central region who reached the 
Proficient level was higher than the percentage in the 



Southeast In addition, when compared with children from 
the Southeast and Northeast a higher percentage of 
Central region students were able to reach the Basic 
level. There were no significant regional differences in 
attainment of fht Advanced level. 

At grade 8, a somewhat different pattern was evident 
The percentages of students in the Central and Northeast 
regions who reached the Proficient and Basic levels 
were higher than the percentages for the Southeast and 
West The percentage of students at the level 

was higher in the Central region than in the Southeast 

A smaller percentage of high school seniors reached 
the Basic level in the Southeast than in the rest of the 
country. In addition, the percentage of students in the 
Southeast who reached the Proficient level was lower 
than the percentages in the Central or West regions. At 
grade 12, there were no statistically significant r^onal 
differences in the attainment of the Advanced level. 
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Geography Achievement Levels for 
^lajor Reporting Subgroups 

In this section, variations in performance among the 
major reporting subgroups are discussed. Again, the 
discussion is confined to those differences that were 
statistically significant 



Race/Etimicity^ Tbble 32 presents achievement level 
results for students in six racial/ethnic groups. Consis- 
tent with past assessments, results from the NAEP 1994 
geography assessment indicated differences in perfor- 
mance among racial/ethnic subgroups. Significant 
differences among racial/ethnic groups were observed in 
the percentages of students at or above each of the three 
achievement levels: Basic, Proficient, and Advanced. 
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At grade 4, the percentages of White students who 
attained each of the three achievement levels were 
higher than the percentages for Black and Hispanic 
students. Furthermore, higher percentages of White 
students reached the Basic and Proficient levels than did 
American Indian students. Higher percentages of Asian 
students than Black and Hispanic students reached the 
Basic and Proficient levels. Also, the percentage of Asian 
students at or above the Proficient level was higher than 
that of American Indian students. The percentage of 
Pacific Islander students who reached the Basic level 
was higher than that of Black and Hispanic students. 

The percentages of Hispanic and American Indian 
students who reached the Basic level were higher than 
the percentage of Black students. Finally, a higher 
percentage of Hispanic students than Black students 
reached the Proficient level. 

At grade 8, the pattern shown at grade 4 was largely 
repeated. However, the sample at grade 8 does not allow 
for accurate determination of the standard errors 
associated with the percentages for Pacific Islander and 
American Indian students. Therefore, these groups 
cannot be compared with other groups at this grade. 
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At grade 12, fewer significant differences existed 
across the racial/ethnic subgroups than at the other 
grades. For example, there were no significant 
differences among racial/ethnic subgroups in the 
percentages of students reaching Advanced level. 

The percentages of White and Asian students who 
reached the Basic and Proficient levels were higher than 
those for Black and Hispanic students. Finally, a lower 
percentage of Black students than Hispanic students 
reached the Basic level. As at grade 8, Pacific Islander 
and American Indian students are not included in group 
comparisons at grade 12 because of sample limitations. 

When considering these data, readers should keep in 
mind the cautions about interpreting group differences 
discussed in Chapter 1. A further reason for caution is 
that, while the percentages of White students scoring at the 
Advanced level were larger (in a statistically significant 
sense) than the percentages of Black or Hispanic 
students at grades 4 and 8, the practical importance of 
these differences may be limited by the small numbers 
of students reaching the Advanced level. 
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Gender. Table 3.3 presents achievement level results for 
male and female students. At grades 4 and 8, the 
significant gender differences observed in average scale 
scores were also observed in the percentages of students 
reaching the Proficient w\A Advanced achievement 
levels. For excunple, the percentages of male students at 
or above the Proficient level (26 percent at grade 4 and 
30 percent at grade 8) were significantly higher than the 
percentages for female students (19 and 25 percent, 
respectively). 



At grade 12, the achievement level results also showed 
a significant gender difference, with male students 
outperforming their female peers. The percentages of 
male twelfth graders at or above the Basic and Proficient 
achievement levels were higher than the percentages of 
female students. Nearly three-quarters (73 percent) of 
male high school seniors reached the Basic level, 
compared with two-thirds (67 percent) of female high 
school seniors. 
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Parents* Highest Level of Education. As shown in 
Table 3.4, a positive relationship exists between level of 
parents’ education and the percentages of students at 
or above the three achievement levels. In general, the 
higher the level of education reported, the higher the 
percentages of students at each achievement level. 



At all three grades, a higher percentage of students 
who reported that at least one parent had graduated 
from college reached the Advanced level than did those 
reporting that at least one parent had graduated horn 
high school. In addition, at grade 8, a higher percentage 
of students who reported that at least one parent had 
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graduated from college reached the Advanced level 
than did those reporting all other levels of parental 
^ucation. 

At the Proficient and Basic levels, the patterns were 
similar. At grade 4, higher percentages of students who 
reported that at least one parent graduated from college 
or had some education after high school reached the 
Basic and Proficient levels than did those reporting 
that their parents had a high school or lower level of 
education. (At grade 4, the percentage of students 
reaching the Proficient and Basic levels were not 
signihcantly different for the groups of students 
reporting the two higher education levels or for the 
groups reporting the two lower education levels.) 

At grades 8 and 12, higher percentages of students 
reporting that their parents had achieved a given level of 
education reached the Basic and Proficient achievement 
levels than did those reporting lower levels of parental 
education. For example, at grade 12, higher percentages 
of students who reported that at least one parent grad- 
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uated from college scored at or above the Basic and 
Proficient levels than did students who reported that at 
least one parent had some education after high school. 
Higher percentages of the latter group scored at or 
above the Basic and Proficient levels than did students 
who reported at least one parent graduated from high 
school; these students, in turn, showed higher 
achievement than those who reported that neither 
parent had finished high school. The exception to this 
rule was at grade 8, where there were no significant 
differences in the percentages of students at or above the 
Basic level who reported that at least one parent graduated 
from college and those who reported that at least one 
parent had some education after high school. 

One-third of fourth graders and one-tenth of eighth 
graders did not know their parents’ level of education. 

In addition, as noted previously, the accuracy of student 
self-reported data may be open to some question. None- 
theless, the positive relationship between parental 
education and achievement in geography remains striking. 
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T^fpe ofLocation» Ictble 3.5 presents achievement level 
results by type of location. T^e of location categories 
indicate the location of students’ schools and are r t 
intended to indicate or imply social and economic 
meanings. (The type of location classifications are 
described iri'^pendix A.) 

At all thre^ grades, no statistically significant 
difierences existed among percentages of students 
attending schools in different types of location who 
attained Advanced level. At grades 4 and 8, higher 
percentages of students attending schools in urban 
fiinge/large town locations than students in central city 



locations reached the Proficient level. At grades 4 and 8, 
higher percentages of students attending schools in urban 
fringe/large town and rural/small town locations than 
students in central city locations reached Basic level. 

At grade 12, a higher percentage of students 
attending schools in urban fringe/large town locations 
than those in rural/small town locations reached the 
Proficient level. Finally, the percentage of urban fringe/ 
large town students who reached the Basic level was 
higher than the percentages of students in other types 
of location. 
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Title / Participation, l^le 3.6 provides achievement 
level results by Title I status. Compared with their 
counterparts who did not receive Title I services, lower 
percentages of fourth-grade Title I recipients performed 
at or above the Basic and Proficient achievement levels. 
Significantly lower percentages of eighth-grade students 
who received Title I services performed at or above each 
of the achievement levels when compared with their 
counterparts who did not receive Title I services. The 



percentages of Title I students who performed below 
Basic were 71 percent at grade 4 and 62 percent at grac 
8. Conversely, between 24 and 27 percent of students 
who did not receive Title I services performed below th( 
Basic level. Grade 12 differences are not discussed here 
because the nature of the grade 12 sample does not 
allow for accurate estimation of the variability of the 
percentages for Title I recipients. 
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Type of School, Table 3.7 presents results for public and 
nonpublic schools. At each grade, the percent^es of 
nonpublic school students who reached both the Basic 
and Proficient levels were significantly higher than the 
percentages of students attending public schools. At the 
eighth grade, a higher percentage of nonpublic school 
students reached the level. 



As noted in Chapter 1, the reader is cauticxied against 
making causal inferences about the relative effectiveness 
of public and nonpublic schools. Achievement level 
differences between the two types of schools are, in part, 
related to socioeconomic and sociological factors, such 
as levels of parental involvement in their child’s educa- 
tion. To get a clearer picture of the differences between 
public and nonpublic schools, more in-depth analyses 
are required. < 
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Summaiy 

For the most part, performances of reporting subgroups 

reflected patterns of average scale scores noted in 

Chapter 2 of this report 

>■ For the nation as a whole, approximately 70 percent 
of students reached the Basic level, while approxi- 
mately one-quarter reached the Proficient level. 

► Higher percentages of White and Asian students 
reached the Basic and Proficient levels than did 
Black and Hispanic students, at all three grades. 

► At grades 4, 8, and 12, the percentages of male 
s^dents who reached the Proficient level were 
higher than those of female students. 






► Generally, the higher the level of parental educatio 
the greater the percentages of students reaching a 
given achievement level. 

► Similarly, higher percentages of students attending 
nonpublic schools at all three grades reached the 
Basic and Proficient levels than did those attending 
public schools. At grade 8, a higher percentage of 
students attending nonpublic schools also reached 
the Advanced level than did their peers who were 
attending public schools. 
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Contexts in Which Students 
Leant Geography 

Geography learning takes many forms. Teachers of 
geography employ a variety of teaching strategies and 
use different geography tools, such as maps, globes, and 
computers, to enhance geography learning. Outside of 
the classroom, exposure to different media and home 
support for schooling can also promote a rich geography 
education.* 

Previous chapters in this report have examined what 
students know and can do in geography. In this chapter, 
some of the contexts in which geography learning takes 
place are examined, to provide a background for under- 
siding student performance. 

The Extent of Students’ Social 
Studies and Geography Instruction 

As part of the NAEP 1994 geography assessment, 
students were asked a series of questions concerning the 
amount of geography instruction they had received. In 
addition, information concerning social studies or 
geography instruction was collected from the teachers 
of fourth-grade students participating in the : v, 
assessment* The results illustrate the nature of 
geography instruction in the nation's schools. 

'■ V': ‘ A y , . 

Social Studies and Geography Instruction at Grade 4. 
The NAEP 1994 geography framework that served as 
the blueprint for the assessment stated that most fourth 
graders will receive some geography education as part of 
their social studies classes, instead of specihc classes in 
geography. Approximately three-quarters (72 percent) of 
fourth graders reported having a social studies class at 
least three times a week. Only 6 percent reported never 
or hardly ever having such classes. Tbble 4.1 shows the 
percentages of students and average scale scores 
according to the frequency with which students 
reported taking i^al studies classes. 

The frequency of social studies classes is positively 
related to fourth graders’ performance on the NAEP 
geography assessment. Students who reported having 
social studies class every day or three to four times a 
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week had higher average scores than those who report* 
having such classes less frequently or never. In additioi 
students who reported having classes once or twice a 
week outperformed those who reported never cr hardly 
ever taking a social studies class. 

Teachers of fourth graders were asked how much 
time they spent on geography in their social studies 
classes. Thble 4.2 shows that teachers of 39 percent of 
fourth-grade students reported spending 60 minutes or 
more per week on geography, while teachers of 44 
percent of fourth graders reported spending 30 minutes or 
less per week on geography. No significant differences in 
average scorw were evident among students from 
classes with different amounts (^tiine spent on geography. 
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Examined together, the results of the NAEP 
questions concerning fourth-grade students’ in-school 
exposure to geography indicated: 

m 

► The majority of fourth graders reported taking 
social studies classes. 

► Frequent social studies classes are positively related 
to scores on the NAEP geography assessment, 
although teachers of 44 percent of fourth graders 
reported spending 30 minutes or less per week on 
geography. 



Geography Course Taking at Grades 8 and 12. Eighth 
and twelfth graders were asked whether th^ had 
previously taken or were currently taking a geography 
course. In addition, eighth graders were asked whether 
they had taken an earth science course, and twelfth 
graders whether they had taken courses in world 
geography and United States geography. 

Table 4.3 presents the results of these questions. 
Fifty-three percent of eighth graders indicated that 
they were currently taking a geography course, and 
69 percent indicated that they had taken at least one 
geography course since the sixth grade. Sevenfy-three 
percent of eighth graders also reported that they had 
taken an earth science course. 

The relationship between course taking and 
geography performance at grade 8 indicates that 
students who reported taking at least one geograq)hy 
course scored significant^ higher than students who 
reported not taking any geography courses. (Thirteen 
percent of eighth graders reported not knowing whether 
they had taken a geography course in the past) 
Furthermore, studente who rei^rted taking an earth 
science course (which c^'rs <» similar to physical 
geography) outperfo^ ^dents who reported that 
th^ had not taken sudi a cpuree. (There was no 
significant difference between ei^th graders who reported 
they were currently taldng a geography course and those 
who were not) 

The majority of twelfth graders (79 percent) reported 
that they were not currently taking a geography course. 
These students, on average, ouh>erformed the 15 percent of 
the twelfth graders who reported they were currently 
enrolled in a geography course. (Again, some twelfth 
graders reported not knowing whether they were currently 
taking a geography course.) At the twelfth grade, the 66 
percent of students who reported taking at least one 
geography course since the ninth grade did not differ 
significantly in performance from the 31 percent who 
reported not taking geography. 
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One explanation for this the significant difference 
between those who are currently enrolled ih a geog- 
r.'^hy course and those who are not is that students 
enrolled in a geography course in the twelfth grade are 
not following course-taking patterns typical of most of 
their peers. This difference in course-taking patterns 
may indicate other academic differences between the 
two groups. However, twelfth graders who reported 
taking a course in world geogrz^hy outperformed those 
students who had not There were no significant differ- 
ences between students who reported taking a course in 
United States geography and those who had not 

Perhaps more consistent with the grade 12 
performance for students who reported taking world 
geography, Thble 4.4 shows that a positive relationship 
exists between the numbers of semesters of potential 
geography-related course work completed by twelfth 
graders and their average scale scores. The results show 
that as the number of semesters of geography-related 
course work increases, average scale scores also 
increase. 
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Geography Course Taking Within Subgroups at 
Grades 8 and 12. Chapter 2 discussed significant 
differences at grades 8 and 12 among various subgroup 
including males and females, racial/ethnic groups, and 
students attending public and nonpublic schools. One 
possible explanation for such differences could be 
variations in course-taking patterns that are measures 
of exposure to geography instruction. Thbles 4.5 and 4. 
present results for selected course-taking variables. 

At grade 8, no significant differences in course 
taking were observed between male and female student 
The percentage of White students who reported taking 
at least one geography course since the sixth grade was 
si^ificMtly higher than the percentages for Black and 
Hispanic students (see Table 4.5). Also, the percentages 
of students attending nonpublic schools who reported 
taking at least one geography course and who reported 
taking an earth science course were greater than the 
percentages of their peers attending public schools. 
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At grade 12 , the only significant difference between 
male and female students was that more male students 
than female students reported taking a United States 
geography course. The percentage of Black students 
who reported taking at least one geography course was 
significantly higher than the percentage for Hispanic 
students (see Ibble 4.6). Also, the percentage of White 
students who reported taking a world geography course 
was significantly higher than the percentage of IWspanic 
students. The percentage of students attending public 
schools who reported taking at least one geography 
course in high school was larger than the percentage for 
students attending nonpublic schools. 
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Differences in course-taking behaviors is only 
one possible explanation for differences in student 
performance on the NAEP geography assessment. 

Many other variables could explain the observed score 
differences. Moreover, not all of the significant group 
differences discussed in Chapter 2 were accompanied by 
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corresponding differences in course-taking behaviors 
(e.g., gender differences at grades 8 and 12). Hence, the 
reader is cautioned against making causal inferences in 
regard to the relationship between course taking arid 
performance on the NAEP geography assessment 
Nevertheless, the breadth of the NAEP geogr^hy data 
affords other researchers the opportunity to explore 
these and other factors and their relationships to 
geography achievement ' ^ 



Teacher Background and Experience 



Given the current interest (discussed in Chapter 1) in 
making geography learning a substantive part of 
educational curricula, it is useful to explore the 
educational backgrounds and experiences that teachers - 
of students in grades 4 and 8 bring to their geography 
lessons. As part of the NAEP assessment teachers were 
asked a series of questions about their educational 
backgrounds. Teachers of assessed students in grades 4 
and 8 were asked about their undergraduate and 
graduate fields of study and about the kinds of 
continuing education they had pursued. (Fourth- 
and eighth-grade teachers were asked to indicate all 
undergraduate and graduate majors that applied. 
Therefore, some teachers indicated two or more majors, 
so that the percen^es in Ihbles 4.7 and 4.8 sum to 
more than 100.) ' 
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The results suggest that teachers of eighth graders 
may have more subject-specific backgrounds, while 
teachers of fourth graders are more likely to have 
Backgrounds specializing in education. It should be 
noted that the organization of fourth- and eighth-grade 
classes also differs. According to school administrators, 

56 percent of fourth graders attend schools where the 
fourth-grade classrooms are described as self-contained 
(students have the same teacher for all academic 
subjects). At grade 8, the majority of students 
(85 percent) attend schools where the organization of 
eighth-grade classes is described as departmentalized 
(students having different teachers for most or all 
academic subjects). The difference in classroom 
organization may explain the differences in teachers’ 
educational bacl^rounds. Thble 4.7 shows teachers of 
most fourth-grade students (80 percent) reported 
having majored in education at the undergraduate level, 
while teachers of 46 percent of eighth graders reported a 
major in education. However, teachers of approximately 
half of eighth-grade students reported having an 
undergraduate degree in either history or a social 
studies education. Few students at either grade had 
teachers who reported a major in geography (2 percent 
at grade 4 and 9 percent at grade 8). 



Similarly, at the graduate level (Thble 4.8), teachers 
of the majority (59 percent) of fourth graders reported 
having a graduate degree in education, while few fourth 
graders had teachers who reported having graduate 
degrees in either geography, history, or social studies 
education. Teachers of 47 percent of eighth-grade 
students had degrees in education. Noticeably more 
eighth graders had teachers who reported graduate 
majors in history and/or social studies education. Again, 
few students at either grade had teachers who reported a 
graduate major in geography. (About 20 percent of both 
fourth and eighth graders had teachers who reported 
other graduate majors, and approximately another 
20 percent reported no graduate study.) 

Tbble 4.9 shows continuing education activities. 
Results indicate that eighth graders had teachers who 
attended somewhat more workshops or seminars than 
did teachers of grade 4. At grade 4, teachers of 
35 percent of students reported participating in no 
continuing education activities in subjects related to 
social studies, history, or geography compared with 
18 percent at grade 8. TVventy-four percent of grade 4 
students had teachers who reported spending at least 
6 hours in workshops or seminars, compared with 
54 percent at grade 8. . . ; ; * . ; 
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The Context for Learning Geography 
at.Grades 4 and 8 

Activities in school, family support for students’ 
educational efforts, and student motivation create the 
context in which academic learning takes place. Hence 
this section contains contextual information related to 
instructional activities in, home support for, and student 
interest in geography learning. Because of the small 
percentage of twelfth graders currently taking a 
geography course (15 percent), grade 12 students’ 
reports of instructional activities related to geography 
are not presented. (The small percentage of twelfth 
graders who reported currently taking a geography 
course may be due to school requirements for tiie 
subject. Sixty-sfac percent of grade 12 students attended 
schools that reported having no geography requirement 
for graduation.^) Teachers’ reports of instructional 
activities for ei^th graders are not presented because of 
limitations of the data (see Endnote 2). 



Instructional Materials and Practices 

Teachers use a range of instructional materials and 
practices in their classrooms. Studies have shown that 
incorporating a variety of teaching methods and 
materials can help eir^ge students with different 
learning styles.* Moreover, the many applications of 
geographic knowledge/ahd the varieties of subject 
matter included in geo^i^hy learning, make the use of 
different materi^ wd ticking practices especially 
important.^ In addition to stwdard textbooks, teachers 
may use maps and ^obes, special projects, and 
computers as part of their instruction. > 






As presented in the following discussion, some 
discrepancies exist between student- and teacher- 
reported hrequencies for some of the instructional 
materials and practices examined. It is not possible to 
offer conclusive reasons for these discrepancies or to 
determine which report most accurately reflects fourth- 
grade classroom activities. The reports presented 
represent students’ and teachers’ impressions of the 
frequency of various activities in their classrooms. 

Instructional materials and strategies m<^ be 
chosen for a variety of purposes. Consequently, the 
relationships between frequency of-use of various 
materials and practices and scores on the NAEP 
geography assessment cannot be interpreted in a causal 
frshion. Ihe design of the assessment does not allow for 
the evaluation of the effectiveness of different strategies. 
The initial ability of the students, the particular topics ' 
being taught, and the complexity of the subject matter 
all need to be controlled before cause-and-effect 
statements can be made, and such experimental 
methods were not possible in a large-scale assessment. 
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The Use of Maps and Globes in Learning Geography. 
^ 3 ny of the geography skills that students are expected 
fo master involve the use of special tools and materials.® 
Maps and globes are the primary tools of geography 
because they assist in the visualization of space and can 
be used to communicate complex social and physical 
data,’' As part of the NAEP 1994 geography assessment, 
students in grades 4 and 8 and teachers of fourth-grade 
students were asked about the frequency of using maps 
and globes in school. Thbles 4.10 and 4.11 present 
teachers’ and students’ reports on the use of maps and 
globes in geography instruction. 

Tbble 4.10 shows that fourth-grade teachers 
reported higher frequencies of use of maps and globes 
than did their students. Teachers of 83 percent of foui^ 
graders reported use of these tools at least once or twice 
a week. Teachers of only one percent of fourth graders 
reported never using maps and globes in class. The 
grade 4 students who had teachers reporting daily and 
weekly use of maps and globes outperformed students 
who had teachers reporting monthly use. 
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As Tbble 4.11 shows, 50 percent of fourth graders 
and 39 percent of eighth graders reported using maps 
and globes at least once or twice a week. TVventy-seven 
percent of fourth graders and 28 percent of eighth 
graders reported never using maps and globes. 

Students in grade 4 who reported using maps and 
globes once or twice a month outperformed students 
reporting use more frequently, and students reporting 
use once or twice a week outperformed students 
reporting use almost every day. Students reporting use 
once or twice a week or once or twice a month out- 
performed students who reported never using maps 
and globes at grade 4. Furthermore, at grade 8, students 
who reported any use outperformed students reporting 
no use. 
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The Use of Projects in Learning Geography, 

Geography projects can help students learn to acquire 
information from primary and secondary sources and to 
andyze, synthesize, and evaluate geographic informa- 
tion, all important skills discussed in the NAEP 1994 
geography framework.® Thbles 4.12 and 4.13 show the 
percentages and average scale scores for teachers’ and 
students’ reports of doing projects related to geography 
study. Thble 4.12 shows that at the fourth grade, teacher- 
reported frequencies of never using projects are rela- 
tively consistent with student-reported frequencies. 
Teachers of 82 percent of fourth graders reported having 
students do projects related to geography at least 
sometimes. Teachers of 18 percent of fourth graders 
reported never having students do such projects. There 
were no significant differences in student performance 
by frequency of using projects. 
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As Thble 4.13 indicates, 78 percent of fourth graders 
and 69 percent of eight giraders reported dding ’ 

geography projects at least sometimes. TWenty-three 
percent of fourth graders and 31 percent of eighth (? ' 
graders reported never doing such projects. 

As with the use of maps and globes (see Tbble 4.11), 
there was some relationship between students’ reports 
of doing projects related to geography and geography 
performance on the assessment. At grade 4, students 
reporting doing such projects often or sometimes out- 
performed students who reported never doing such 
projects; and at grade 8, students reporting doing projects 
sometimes outperformed students who never did them. 

When reported use of projects for the 53 percent of 
eighth-graders currently enrolled in a geography course 
is examined, the picture changes. Thble 4.14 shows the 
results for eighth graders’ reports of doing geography 
projects only for those grade 8 students who reported 
they were currently taking a geography class. (The 

ERIC 



TABir4.13 


1 SlMhuts' R«Mrts N Dohg 

G^ptyPraj^ ~S 

Grades 4 ondS taaussss. 


— ■ ' 




6ra4(4 


6n4t8 


PmmIv| 8 mA 

Awift Sciii Scifo 


AMVipi Sciii Scofo 


OftM 

ScmetiiMS 

Him 


29(1.1) 

211 (1 J) 
49(0.9) 
208(1.6} 
23(0.8) 
198(1.8) 

1 


. 21 (0.8) 
..260(1.1) 
48(0.8) 
262(0.9) 
31 (1.0) 
257(1.2) 


MkiMti bitVMi Ai snips My U pvliily IX|M 
Tk sImM iirm il Ai isHndiJ pirnilisis fwigi sob snm ipM 

nii idA fS-pifisM cMtAGy thtf, fir mA icpilollM if Miml, in 
vlAii plu ir «iMi IM sImM strm if M istiMPi ^ 

PPrciM|« if tfiGiili ii At si^nipt MV Ml Mri too Am ^ 

SOUtg: MmH (MUr fir EActHm Sirtsito, MiiHuri KimmiiK if CiiMlnrf Rngnc {UOl 
t 0 f 4 GHgrapfir AssmmmI. 



eighth-grade results presented in Thble 4.13 are for all 
eighth graders. Reports of geography-related projects by 
students not currently enrolled in a geography class 
may be associated wi^ projects assigned in history or 
social studies classes.) For those eighth-grade students 
who reported they were currently taking a geography 
course, the frequencies of doing geography-related 
reports were similar to those reported by all eighth 
graders (see Tbble 4.13). However, unlike the results for 
all eighth graders that show some relations)^ between 
doing geography reports and performance, n<;j'slgnificant 
differences were observed for the subset of iatiilents 
currently taking a geography course. • 
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The Use of Computers in Learning Geography* 
Computers can be an important classroom tool. In 
geography learning, they have many uses, including 
developing models of complex geographic relationships 
and environmental and cultural processes.® Tables 4.15 
and 4.16 present teachers’ and students’ reports on the 
frequency of use of computers for studying geography. 

Teacher-reported frequencies of computer use 
were more or less consistent with student-reported 
frequencies. Ta)le 4.15 shows that teachers of 42 percent 
of fourth graders reported using computers at least 
sometimes. As with student-reported data, teachers of 
the majority of fourth graders reported never using 
computers in the teaching of geography. 

There was a somewhat different relationship between 
use of computers and student performance associated 
with fourth-grade teachers’ reports than that associated 
with student-reported data; grade 4 students who had 
teachers reporting using computers sometimes outper- 
formed students who had teachers reporting never 
using computers. (Students whose teachers reported 
using computers sometimes also outperformed those 
whose teachers reported using computers often.) 
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As Tbble 4.16 indicate, about 30 percent of both 
fourth- and eighth-grade students reported using 
computers at least sometimes for geography study. Mor 
than two-thirds of students at both grades reported not 
using computers for geography: 68 percent at grade 4 
and 70 percent at grade 8. Fourth-grade students 
reporting using computers sometimes or never 
outperformed students reporting using them often. 
(Also, fourth graders who reported never using a 
computer outperformed those who reported sometimes 
using them.) There were no significant differences at 
grade 8. Again, given the number of variables (such as 
student ability levels and different uses of computers) 
that the NAEP geography assessment cannot account 
for, the reader is asked to avoid inferring simple ca usal 
relationships from these results. 
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The Use of Films, Videos, or Filmstrips in Learning 
Geography. Audiovisual material can communicate and 
enhance many kinds of information important to the 
stu'dy of geography. For example, major geographic 
processes, such as ocean currents and vulcanism, can be 
vividly illustrated with such materials, as can aspects of 
life in different cultures and communities. Thble 4.17 
presents fourth-grade teachers’ reports about the 
frequency of use of these materials for studying 
geography, and Thble 4.18 presents fourth- and eighth- 
grade students’ reports. 

As Thble 4.17 shows, teachers of most fourth graders 
(79 percent) reported using films, videos, or filmstrips 
at least sometimes. The average scale scores for students 
whose teachers reported some use of these materials was 
higher than that of students whose teachers reported never 
using them. 
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Thble 4.18 shows that 65 percent of fourth graders 
and 69 percent of eighth graders who reported use of 
audiovisual materials at least sometimes. Approximately 
one-third of both fourth- and eighth-grade students 
reported never using such materials, while teachers of 
only 21 percent of fourth-grade students reported no use. 

There was a positive relationship between average 
scale scores and the use of films, videos, and filmstrips. 
At both grades 4 and 8, students who reported that these 
materials were used sometimes outperformed those 
who reported that these materials were not used, and 
at grade 8, students who reported that such materials 
were used often also outperformed those reporting they 
were not used. 
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The Us0 ofHomewwrk in Learning Geography* 
Teachers frequently reinforce the lessons taught in their 
classrooms with homework. As part of the NAEP assess- 
ment, students at all three grades were asked to indicate 
the amount of time they usually spend on homework 
each day. This question was not restricted to social 
studies or geography homework but referred to home- 
work across all subject areas. Thble 4.19 presents the 
results for fourth, eighth, and twelfth graders. Overall, 
a positive relationship existed between spending some 
time on homework and average geography scale scores. 

At all grades, between 7 and 14 percent of the 
students indicated that they did not usually have any 
homework assigned. For students who indicated that 
they usually had homework assigned, those reporting 
that they spent some time on homework significantly 
outperformed those who reported they did not usually 
do their homework. (At grade 12, the average scale 
scores for student who usually don’t do homework and 
those who spend an hour were not significantly 
different.) At grade 4, the average score for students who 
reported spending more than one hour on homework 
each day was significantly less than that for those who 
reported spending one hour. At grade 8, average scale 
scores increased with each incremental increase in the 
time spent on homework. At grade 12, the average scale 
score for students who reported spending more than 
one hour on homework each day was significantly 
higher than that for students who reported spending 
one hour. 






Eighth graders were also asked about how much 
geography homework they did each week. Tbble 4.20 
presents the results for students who reported that 
they were currently taking a geography class. Students 
who reported spending 1 or 2 hours on geognq)hy 
homework performed significantly better than those 
reporting no time spent on geography homework. 
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The Use ofNewspaperSt Magazines f or Journals in 
Learning Geography, Articles in a range of periodicals 
can supplement geography learning both inside and 
outride the classroom.^** Students who indicated that 
they do read such materials may be reading them both 
in and out of school. In either case, for eighth-grade 
students, there was a positive relationship between 
reading these materials and geography performance. 
Ibble 4.21 shows percentages of students reading 
periodicals related to geography and their average 
scale scores. 

Approximately two-thirds of students at grades 4 
and 8 reported reading such materials at least some- 
times. At grade 8, students who reported reading these 
materials sometimes outperformed students who 
reported never reading them. However, at grade 4, 
students who reported reading such periodicals some- 
times or never outperformed those who reported 
reading them more frequently. 
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Students’ Home Support 

'Kie support students receive at home for learning is as 
important in geography as it is for other subjects 
surveyed in NAEP assessments. Variables related to 
students’ home environment often show a strong 
relationship to performance. The following section 
presents information about home support for geography 
learning. 

Discussing Schoohoork at Home. A measure of the 
importance of schoolwork for students and their 
families is how often schoolwork is discussed at home. 
When students discuss academic work at home, they 
create an important link between home and school. 
Recent studies have noted the positive relationship 
between parental involvement in schooling and student 
achievement.” Supporting cooperation between parents 
and schools is a major objective of many recent edu- 
cational reform efforts, among them the National 
Education Goals.^ 

Students in the NAEP 1994 geography assessment 
were asked how frequently they discussed their studies 



at home. (This question was not restricted to discussing 
social studies or geography studies at home but referred 
to all subject areas.) Their responses are summarized 
in Thble 4.22. The majority of students across grades 
reported discussing their studies at home at least 
sometimes. Seventy-eight percent of fourth graders, 

68 percent of eighth graders, and 62 percent of twelfth 
graders reported discussing their studies at home at 
least once or twice a week. However, approximately 
one-fifth of fourth and eighth graders, and nearly one- 
quarter of twelfth graders, reported never or hardly ever 
discussing their studies at home. 

At all three grades, students who reported never 
or hardly ever discussing their studies at home were 
outperformed by students who reported discussing 
their studies on a regular basis. At grade 8, students 
who reported daily discussions outperformed students 
who reported discussions once or twice a week or 
month, and students who reported discussions once or 
twice a week outperformed those reporting discussions 
once or twice a month. At grade 12, students who 
reported daily or weekly discussions outperformed those 
reporting discussions once or twice a month. 
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Literaqf Matertab in the Home* Students can leam 
much about geography hrom written sources read 
outside of school. Newspapers and magazines, for 
example, can expand students’ knowledge of place 
locations, as well as cultural, environmental, and 
economic patterns relevant to today’s geography 
curriculum. Moreover, parents who read regularly 
model a pattern of curiosity and lifelong learning that 
is important for their childrens’ academic success.’^ 

As part of the NAEP assessments, students at all three 
grades are asked whether their families have an 
encyclopedia, receive a newspaper regularly, receive any 



magazines regularly, or have more than 25 books in the 
home. 'Ibble 4.23 presents the percentages of students 
reporting that their families have all four types, only 
three types, or two or fewer types of these literacy 
materials. 

As presented in the table, 38 percent of fourth graders, 
51 percent of eighth graders, and 57 percent of twelfth 
graders reported having all four kinds of literacy materi- 
als in the home. Across all three grades, the more types 
of literacy materials reported in the home, the higher 
the average geography scale scores. 
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Television Viewing Habits, Past NAEP assessments 
have highlighted the national concern over the amount 
of time students spend watching television. For 
example, the NAEP 1992 and 1994 reading assessments 
found a negative relationship between the amount of 
television watching and reading performance. A major 
concern is the possibility that time spent watching 
television may be consuming time that could be spent 
on schoolwork or reading activities. 

Tbble 4.24 presents students’ reports of their 
tehvision-viewing habits. Thirty-nine percent of fourth 
graders, 44 percent of eighth graders, and 46 percent of 
twelfth graders, reported watching two to three hours of 



television per day. TWenty percent <rf fourth graders, 

16 percent of eighth graders, and 7 percent of twelfth 
graders reported watching six hours or more per day. 

Generally, there was a negative relationship between 
increased hours of television viewing and student 
performance. At all three grades, students who reported 
watching six or more hours of television per day scored 
lower, on average, than students who reported less 
frequent viewing. In addition, at grade 4, students who 
reported watching two to three hours of television per 
day outperformed those who watched four to five hours 
per day. At grades 8 and 12, the less time spent watching 
television, the higher the average geography score. 
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Student Interest in Geography 

A curiosity about and interest in geography may foster 
learning of the subject at school or at home. The NAEP 
1994 geography assessment asked students to indicate 
whether geography is their favorite subject, whether 
they like other subjects better, or whether they have 
never studied geography. Tbble 4.25 shows student- 
reported degrees of interest in geography, and related 
average scale scores. 



As the table indicates, between 14 and 26 percent of 
students reported that geography was their favorite 
subject. Students across the three grades who did report 
that geography was their favorite subject scored signifi- 
cantly higher than students who preferred other subjects. 



F ' 
[■ 

\y 



tABlE4.25 



Students' Reports on How Much 
They Like Studying Geography 
Grades 4, 8, ond 12 
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Summary 

The picture of geography education and student 
performance gained from an examination of background 
variables relating both to course work and classroom 
practices is quite varied. Encouragingly, exposure to 
geography at grade 8 and to social studies at grades 4 
and 12 was associated with higher average scale scores. 
While grade 12 students who indicated that they were 
currently taking geography had lower scores, the more 
semesters of potentially geography-related course work 
done, the hi^er the scores. Moreover, grade 12 students 
who indicated that they had taken a course in world 
geography outperformed those who reported that they 
had not. 

Although some discrepancies exist between teacher- 
and student-reported data about instructional materials 
and practices, it is fair to say that the materials and 
practices reviewed in this report all received at least 
moderate use. For example, approximately one-third of 
fourth and eighth graders reported using maps and 
globes once or twice a week, while 79 percent of grade 8 
students who were currently taking geography reported 
doing geography-related projects at least sometimes. 



Generally, some use of instructional materials, such 
as films, videos, and filmstrips, was associated with 
higher scale scores, although computer use was 
negatively associated with scores. The relationships 
between amounts of homework (across all subject areas) 
and geography scale scores for students in grades 4, 8, 
and 12 were positive. There was also a positive 
relationship between amount of time spent on 
geography homework and scale scores for grade 8 
students who were currently taking gragraphy. 

Performance associated with home support variables 
was consistent with educational research findings. Dis- 
cussion of schoolwork at home and literacy materials in 
the home were both associated with higiier scores, while 
frequent television watching was associated with lower 
scores. (Given the number of factors the NAEP assess- 
ment cannot control for, the reader is again cautioned 
against drawing causal inferences.) 

Finally, students across the three grades who 
reported tiiat geography was their fevorite subject 
outperformed those who preferred other subjects. 
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CHAPTER 5 




What Students Know and 
Can Do in Geography 

This chapter provides additional perspective on what 
students know and can do in geography by examining 
the specific knowledge and skills demonstrated by 
students at various points on the composite geography 
scale, and the three subscales (the subscales correspond 
to the three geography content areas, around which the 
assessment was organized). 

An overview of the geography knowledge and 
abilities demonstrated by students within three perfor- 
mance ranges on the geography scale, and information 
on their self-repprted study habits, is included. Average 
scale score results are presented for the nation and 
for selected subgroups of students based on the three 
subscales. Finally, selected illustrative questions are 
displayed in item maps, which indicate the types of 
questions that were likely to be answered successfully by 
students scoring at particular levels on the content-area 
subscales. 

Overview of Students’ Performance on 
the NAEP Geography Scale 

The NAEP geography scale, ranging from 0 to 500, 
summarizes the overall score results of students at the 
fourth, eighth, and twelfth grades. The following 
descriptions of students’ knowledge and abilities are 
based on sets of questions that were answered suc- 
cessfully by students performing within three ranges on 
the scale. These ranges represent lower, middle, and 
higher performance based on percentile distributions. 
The sets of questions identified in each of the three 
ranges on the scale were analyzed by geography edu- 
cation experts to characterize the nature of students’ 
geography knowledge and abilities, (/^pendbc B contains 
a description of the procedures lised to generate this 
portrait of students.) 

Fourth-Grade Profits, Grade 4 students who were near 
the 25th percentile (scale range 170 to 187) demon- 
strated success working with maps and diagrams. They 
could read a simple m^ key and an elevation profile. 



and they showed a rudimentary understanding of how to 
construct a map fi-om written directions. They demon- 
strated partial Imowledge of the location of the equator, 
the poles, and continents. These students could identify 
a major ecosystem from visual clues and demonstrated 
an initial tmderstanding of human actions leading to 
deforestation. 

Fourtii -grade students who were near the 50th 
percentile (scale range 206 to 216) were able to extract 
basiic information from an atlas, maps, photographs, and 
a bar graph. Given a number of options on a map, 
students could locate Hawaii. They knew the approxi- 
mate location of their home state or district and could 
locate some of the cr.mtries in North America. They 
showed a rudimentary understanding of how to construct 
a weather map from written directions. Students at this 
level were able to answer some uncomplicated questions 
that did not rely on visual prompts, such as questions 
relating to climate, diffusion of information, and envi- 
ronmental issues such as pollution. 

Fourth-grade students who were near the 90th 
percentile (scale range 249 to 269) demonstrated an 
ability to use scale, measurement, and direction. They 
could interpret and apply information fi-om simple 
thematic maps and graphics. Given outline maps of the 
United States and Canada, these students could identify 
bodies of water and a mountain range. They could also 
identify Canada, the United States, and Mexico and 
could locate Japan and Australia. Students were able to 
construct a weather map and a map of a town fi-om 
written directions. They demonstrated understanding of 
some physical geographic concepts such as Earth-Stm 
relationships and soil erosion. They also demonstrated 
knowledge of the relationship between climate and 
economic activities and limited knowledge of reasons 
for migraiiv)n. These students knew some consequences 
of how technology affects the environment, and they 
had a rudimentary understanding of the consequences 
of human activities, such as waste disposed. 

Eightit-Grade ProGle. Eighth-grade students who were 
near the 25th percentile (scale range 230 to 243) could 
read and extract data from visuals such as diagrams, 
simple line graphs, and a bar graph and could make 
uncomplicated inferences from photographs of desert 
landscapes. They understood a basic coordinate ^tem 
and could identify their home state or district. South 
America, and Antarctica. These students could locate 
information on simple reference maps and, using an 
atlas, could make simple inferences from information 
contained in thematic maps. 








Eighth-grade students who were near the 50th 
percentile (scale range 258 to 267) could locate some 
major rivers and lakes in North America and recognize 
soine simple landform and water features. They could 
construct maps from written instructions with partial 
i accuracy that improved when a grid system was given. 

. They understood the function of a transit map and had 
partial success mapping out a specified route. These 
students demonstrated knowledge of what causes 
earthquakes and were able to make connections between 
I climates and locations of cities and between climates 

i and types of crops. Students were able to recognize 
I relationships between topography and human settle- 
1 ment and could identify a reason for an urban land use 
j pattern. They demonstrated some understanding of 
j movement in relation to immigration and trade. 

Those eighth-grade students who were near the 
90th percentile (scale range 295 to 312) demonstrated 
an ability to use a range of geographic tools such as 
graphs, pie charts, population pyramids, climographs, 
cross-sectional diagrams, and an isotherm map. TTiey 
could identify the largest ocean and knew which country 
was located in the Alps. They could also find major 
physical features on different types of maps at a variety 
of scales. Students could infer geographic processes 
from visual representations, and they could construct 
maps from written instructions using a scale and legend 
j accurately when given a grid and with considerable 
■ accuracy without a grid. These students understood the 
influence of elevation on climate and of climate on 
vegetation and had sorne imderstanding of the effect of 
monsoon winds oh cjtma^ and economy. Students at 
this level demonstrated wme understanding of the 
dynamic interconnections between hunm ^sterns and 
pln'sical systems and understood what con^butes to air 
I pollution. TTiey could apply uncomplicated concepts of 
I economic development, tirade and transportation, and 
economic interdependence and could relate geographic 
concepts to historical knowledge. 



I 




Ticelfth-Grade Profile* Students who were near the 
25th percentile (scale range 259 to 270) could read and 
interpret visual material in the form of single maps, (as 
opposed to sets of maps or maps with accompanying 
text). They demonstrated knowledge of basic physical 
geography terms, could interpret a map to locate a , . 
region, and showed some understanding of processes 
relating to earthquakes and erosion. Students. could 
identify where Spanish is spoken on a world map. These 
students demonstrated a rudimentary understanding of 
relationships between topography and human settle- 
ment and of how relationships between and among 
places are affected by factors such as accessibility, 
proximity, and distance. 

INwelfth-grade students who were near the 50th 
percentile (scale range 283 to 291) were able to locate 
some physical features such as United States deserts and 
identify major world religious centers. They demon- 
strated knowledge of some fundamental geographic 
skills such as the ability to read and interpret contour 
maps, different map projections, and a time zone map. 
Students were able to use atlases to answer fundamental 
physical geography questions relating to climate, ' = » 
natural vegetation, and growing seasons. Th^^ demon- 
strated understanding of how major processes such as ^ 
weathering and erosion shape patterns in the natural 
environment and of the influence of natural systems on 
human environments. Students displayed some ability 
to translate narrative descriptions that include data into 
maps. These students demonstrated knowledge of 
physictd reasons for land use patterns, understending of 
movements such as diffusion and migration, wd some 
understanding of reasons for trade. ■ ^ ' 
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TVvelfth-grade students who were near the 90th 
percentile (scale range 316 to 329) were able to compare 
different kinds of maps, graphics, diagrams, tables, and 
other visuals to draw conclusions and interpret 
information. They were often able to apply some outside 
knowledge to interpret maps and other visuals. These 
students were able to transfer information from one 
reporting form to another and could draw generally 
accurate two-dimensional maps. They could locate and 
label features such as the Mississippi River and the 
Rockies on a cross-sectional diagram of the United 
States, and could identify South America from its 
elevation profile. Students could identify major deserts 
on a world map. They were able to infer some physical 
geomorphic processes from examining physical features 
shown on a map. They also understood regional terms 
such as “megalopolis” and “corn belt”, and concepts 
relating to connections among places such as travel and 
transportation 

Students at this level could recognize the economic 
and political importance of places such as the Suez 
Canal and demonstrated some understanding of how 
political unrest can influence economies. They knew a 
reason for the formation of the Organization of 
Petroleum Exporting Countries and could identify the 
countries that belong to this organization. Using a map 
showing the languages of Africa, students could 
interpret information about language patterns in Africa. 
These students also demonstrated partial understanding 
of causes and consequences of human migration and a 
rudimentary ability to discuss different people’s 
perspectives of the same environment at a variety of 
scales, local to global. 










Profiles of Students’ Geography 
Knowledge, Abilities, and Study Habits 

Figures 5.1, 5.2, and 5.3 show profiles of the lower-, 
middle-, and higher-performing students. The profiles 
link the knowledge and abilities of these students with 
their self-reported study habits, and also summarize the 
performance descriptions from the previous section. 

The study habits presented in the figures are based on 
students’ self-reports about three activities: number of 
pages read per day in school, hours per day spent on 
homework, and frequency of discussing studies at home. 

By examining all three profiles, a common pattern 
emerged at each grade. As the students’ geography 
scores increased, the complexity and sophistication of 
the geographic knowledge and skills they exhibited , 
increased. Moreover, students performing at the higher 
range of the scale were more likely to read more pages 
per day, spend more time on homework, and more 
frequently discuss their studies at home.* 
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Figure 5 J Profiles of bwer, Midde, and Higher Performing Twelfth Graders: Geography Knowledge, Abitieo, and Stidy Habits 
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(ikI otiw visuoh to drm conciu^ and 
faiterprot Information 

^ kxato ond lobe) orange of features of the 
physical and hunmn environrnents on mops 
onddkigroms 

infer some physicolgeomorphk processes 
from physiral features shown on o nwp 

^ discuss in rudimentory terms different 
* people's perspectives of the some 
environment ot o voriety of scoles, 
locol to global 
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Stsdy habits of twelfth graders who 

wore wear the 25th perceatflo: 

^ 29 percent read more than 10 pages eodidoy 
iff school and for homewodt 

^ 52 percent spent one or imrobou5 sod) day 
on homework ' 

^ 55 percent dbcussedstiKfics at home ot least 
once or twice a week 



Tweifth’grade students who ware 

near the SOth percentOo could: 

^ reod and interpret contour mops, iTifferent 
map projections, and a time zone mop 

^ use atlases to onswerfundamentolphyskol 
geography questions, such as those reioting to 
dimtie ond vegetation 

^ show understanding of how major processes, 
such as weathering and erosion, shape potterns 
in the natural environment 

^ show knowledge of major processes of humon 
geography, si^ as migration ond diffusion 



SOth Percentile 




Study habits of twelfth gr aders who 

were near the SOth pcrcaetfli: 

^ 40 percent rood more than 10 pages eodidoy 
in school and for homework 

^ 53 percent spent one or mere hours eodidoy 
onhomework 

rm*-; . 

^ 59 percent dbcussed studies at home ot least 
onceorhriceaweek 



$Oth Percentile 



Study habits of tweKtb grifiwt W 
were near the 90th parcMHb: " 




> f} 



^ 57 percent read more than to pages eodidoy 
in school and for homework 

62 percent spent one or mors hows each day 
onhomework . r:.} 

^ 74 percent ifecussedsfuilos at home dt least 
once or twice a week 
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Average Performance by Content Areas 

I 

I As described in Chapter 1, the NAEP 1994 geography 

j assessment was organized around the following three 

content areas: 

‘ !► Space anrfP/iice. Knowledge of geography related 

to particular places on Earth, to spatial patterns on 
Earth’s surface, and to physical and human 
processes that shape such patterns. 

^ Environment and Society. Knowledge of geography 
related to the interactions between the environment 
and society. 

^ Spatial Dynamics and Connections. Knowledge of 
geography related to spatial variations and connec- 
tions among people and places. 



The three content areas were designed to ensure 
coverage of the major branches of geography. They were 
seen as broad areas that would sometimes overlap to 
accomplish this coverage. (The geographic cognitive 
areas presented in the NAEP geography framework cut 
across all three content areas.) A look at the patterns of 
student performance on the subscales can contribute to 
a more precise understanding of composite scale scores, 
although the patterns of performance among subgroups 
on the composite scale were also generally evident for 
the subscales. 
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Performance on Geography Content Areas for the 
Nation l^le 5.1 presents th&Jl994 average scores by 
geography content areas for students at grades 4, 8, and 
12rThe subscales for the three areas were created 
independently of one another and no attempt was made 
to equate the subscales. Therefore, comparisons across 
subscales (for example, fourth graders performed better 



on Content Area 1 than Content Area 2) are not 
necessarily meaningful. Appendix A contains a more 
extensive di.~cussion of the geography scaling 
procedures. 

Patterns of performance among subgroups on the 
composite scale were also generally evident for the 
content area subscales. 



Averogt Scale Scores ot Various Pe'rcentiles by Geography Content Areos 
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Performance on Geography Contetd Areas by Region, 

Ta)le 5.2 presents the average scale scores and subscale 
results by region. These results are fairly consistent with 
the composite scale results found at each grade. 

> At grade 4, for the composite scale zind all three 
subscales, students from the Central region out- 
performed students from the Southeast. Students 
from the Central region also outperformed students 
from the Northeast for Space and Place, and 
students from the West for Spatial Dynamics 
and Connections. 






► At grade 8, students from the Northeast and Central 
regions outperformed students from the Southeast 
and the West for all three content areas. 

► At grade 12, students from the Northeast out- 
performed students from the Southeast for Space 
and Place and Spatial Dynamics and Connections. 
Central region students outperformed students from 
the Northeast for Space and Place and students 
from the Southeast for all three content areas. 
Finally, stu dents from the West outperformed those 
from the Southeast for all content areas. 




Average Scale Scores In Geography Cooteit Areos by Region 
Grades 4, 8, and 12 
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Performance on Geography Content Areas by Race! 
EOmicUy, I^le 53 reports the subscale results and the 
overall average scale scores for racial/ethnic subgroups. 

some exceptions, White and Asian students out- 
performed Black and Hispanic students, and Hispanic 
students outperformed Black students, as on ^he com- 
posite scale. (Racial/ethnic classificatioriS are based on 
, self-reported informatior. provided by students. ) 

For the Pacific Islaiader and American Indian student 
I samples at grades 8 and 12, the nature of the samples 

does not allow accurate determination of the standard 
; errors. For this reason, dinerences among these samples 

i 



and other racial/ethnic subgroups are not discussed for 
those two grades. 

^ At grade 4, for each content area. White students 
had a significantly higher average score than Black, 
Hispanic, and American Indian students, and 
American Indian students had a significantly higher 
average score than Black students. Hispanic stu- 
dents outperformed Black students for Space and 
Place and Environment and Society, but not for 
Spatial Dynamics and Connections (an exception to 
the pattern for the composite scale). Asian students 
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TSSESsmas: 



outperformed Black and Hispanic students on all 
subscales and American Indian students for Space 
^ ; and Place and Spatial Dynamics and Conmctions. 
Pacific Islander students outperformed Black and 
Hispanic students for Space and Place and Spatial 
Dynamics and Connections and Black students for 
Environment and Society. 

► At grade 8, for each content area. White and Asian 
students had significantly higher average scores than 
Black and Hispanic students. Hispanic students 
outperformed Black students for Space and Place 
and Environment and Society, but not for Spatial 
Dynamics and Connections (again, an exception to 
Hispanic students’ performance on the composite 
scale). 

► At grade 12, for each content area, Hispanic stu- 
dents demonstrated a higher average score 
than Black students, and Asian students demon- 
strated a higher average score than Black and 
Hispanic students. White students had significantly 
higher average scores than Black and Hispanic 
students in all content areas. 



Performance on Geography Content Areas by Gender, 

Thble 5.4 presents male and female students’ perfor- 
mance on the three geography content areas. As 
reported in Chapter 2, males did demonstrate average 
scale scores significantly higher than those for females 
at all three grade levels on the composite scale. There 
were some deviations from this pattern when 
performance was examined at the subscale level. 

^ At grade 4, male students outperforr 3 female 
students for the content area Space w i Place, but 
there were no significant difierences for the other 
two content areas. 

^ At grade 8, male students outperformed female 
students for Space and Place and Environment and 
Society, but not for Spatial Dynamics and 
Connections. 

^ At grade 12, as on the composite scale, male 

students outperformed female students in all three 
geography content areas. 
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Average Stole Scores b Geography Content Areos by Gender 
Grades 4, 8, and 12 
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Performance on Geography Content Areas by Parents* 
Highest Level of Education* l^le 5.5 presents perfor- 
maaice by student-reported parental education level for 
the"three content area subscales. The positive relation- 
ship between parental level of education and performance 
noted with the composite scale was also evident for each 
of the three subscales. Generally, as the level of parental 
education increased, average subscale scores increased. 

^ At grade 4, average subscale scores, for the most 
part, increased with increasing levels of reported 
parental education. However, across subscales, there 
were no significant score differences between 
students who reported that at least one parent had 



graduated hrom college and those who reported that 
at least one parent had some education after high 
school (consistent with the exception for the com- 
posite scale), and between students who reported 
that at least one parent had graduated from high 
school and those who reported that neither parent 
finished high school. 

^ At grade 8, for all three subscales, average subscale 
scores increased with increasing levels of parental 
education. 

► Similarly, at grade 12, for all three subscales, 
average scores increased with increasing levels of 
parental education. 



Averoge Scale Scores in Geography Content Areas 
by Parents' Highest bvel of Edocation 
Grides 4, 8, and 12 
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Peritimumce on Googntphy Content Areas by 1\ipe of 
location, Ta)le 5.6 presents the performance of students 
attending schools in the three types of locations, 
according to the geography content areas. With one 
exception, performances on the three content area 
subscales mirrored performance on the composite scale. 

► At grade 4, for all three subscales, students 
attending schools in urban fringe/large town and 
rural/small town locations outperformed students 
attending schools in central city locations. 



At grade 8, for Space and Place, students attending 
schools in urban firinge/large town locations out- 
performed students attending schools in central 
city lo^tions. (On the composite scale, students 
attending schools in urban fnnge/large town and 
rural/small town locations outperformed students 
attending schools in central city locations.) For thi 
other two subscales, student performance mirrored 
that of the composite scale. 

At grade 12, for all three subscales, students 
attending urban fringe/large town schools 
outperformed students attending both central city 
and rural/small town schools. 
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Average Scale >$ fci Geogropiiy Content Areas by Type of Location 
Grades 4, 8, ond12 
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^ Performance on Geography Content Areas by Type of 
SchooL l^Ie 5.7 presents the performance of students 
attending public and nonpublic schools in the three 
geography content areas. Again, performances on the 
three content area subscales mirrored performance on 



the composite scale. At grades 4, 8, and 12, students 
attending nonpublic schools had significantly higher 
average scores than those attending public schools 
across all three geography content areas. 
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Average Scale Scores in Geography Content Areas by Type of School 
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Profiling Students* Performance on 
the Three Geography Content Areas 

To illustrate the range of assessment tasks that students 
were asked to perform, this section provides a visual 
representation of each geography content area subscale, 
wiled an “item map.” The item map shows a set of tasks 
in the order of their difficulty for the population 
assessed. Each task is identified by a brief description. 
Not all the tasks in the assessment could be presented 
in these figures. Instead, sample tasks were selected to 
illustrate the range of abilities across the subscales and 
to demonstrate the types of tasks that students with 
different subscale scores could typically perform correctly. 

The item maps (Figures 5.4, 5.5, and 5.6) identify 
Ae points, on each content area subscale at which 
individual tasks were answered correctly by 
approximately two-thirds (65 percent) of the students. 
(The criterion was set at 74 percent for multiple-choice 



questions to adjust for the possibility of students’ 
answering correctly by guessing.) The point on the 
subscale item map at which a task is positioned 
represents the subscale score attained by students who 
had a 65-percent probability of successfully performing 
the task. Thus, for each task and its corresponding 
subscale score, it can be said that at least 65 percent of 
students at or above that point on the subscale have 
adequately performed that task. : . ■ 

For example, the fourth-grade Specs and Plocs itenr. 
map (Figure 5.4), indicates that at least 65 percent of 
fourth graders with a score of 216 or better on the 
Space and Place geography content subscale were able 
to locate physical features on a map of North America. 
Fourth graders who scored higher than 216 on this 
subscale were even more likely to demonstrate this 
ability, while students who scored lower on the subscale 
were less likely to do so. In interpreting item map 
information, it should be kept in mind that students at 
different grades demonstrated these geography abilities 
with grade-appropriate materials. 
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Figure 5.4 Mop of Sufuctud Ituns oi Hw Guegrophy 
CootMrt Arta Sobscdes for Grade 4 



I Each geography question was mapped onto the 
■ NAEP geography content area subscale on the 
! basis of students' performance. The point on the 
i subscale at which a question is positioned on the 
map represents the subscale' score attained by 
students who had a 65-percent probability of 
successfully answering the question. 

Thus, it can be said for each question and its 
corresponding subscale score that students with 
i scores above that point on the subscale have a 
I greater than 65-percent chance of successfully 
i answering the question, while those below that 
i point on the subscale have a Ims than 65-percent 
j chance. (The probability was set at 74 percent for 
multiple-choice questions.) 

The item map pr^ented in figure contains 
selected questions to illustrate the as^meht In 
interpreting the item map information, it should 
be kept in mind that students at different grades 
demonstrated these abilities with grade-^propriate 
materials. 
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Figurt 5.5 Map of Sdoctad items on the Geography 
. Coatent Area Siibscaies for Grade 8 



Each geography question was mapped onto the 
NAEP geography content area subscale on the 
basis of students’ performance. The point on the 
subscale at which a question is positioned on the 
nuq) represents the subscale score attained by 
studenf:s who had a 65-percent probability of 
successfully answering the question. 

Thus, it can be said for each question and its 
corresponding subscale score that students with 
scores above that point on the subscale have a 
greater than 65-percent chance of successfully 
answering the question, while those below that 
point on the subscale have a less than 65-percent 
chance. (The probability was set at 74 percent for 
multiple-choice questions.) 

The item map presented in this tigure contains 
selected questions to illustrate the assessment In 
interpreting the item huq> information, it should 
be kept in mind that students at different grades 
demonstrated these abilities with grade-appropriate 
materials. ■ 
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Figure 5.6 Mop of Selected items on the Geography 
CoBteot Area Subscoles for Grade 1 2 



Each geography question was mapped onto the 
NAEP geography content area subscale on the 
basis of students’ performance. The point on the 
subscale at which a question is positioned on the 
; map represents ^scale score attained by 
students who had a 65-percent probability of 

; successfully answering the question. 

, > 

Thus, it can 6e saiid for each question and its 
corresponding subscale score that students with 
scores above that point on the subscale have a 
greater than 65-percent chance of successfully 
answering the question, while those below that 
point on the subscale have a less than 65-percent 
, chance. (The probability was set at 74 percent for 
multiple-choice questions.) 

The item map presented in this figure contains 
selected questiom to illustrate the assessment In 
I interpreting the ite^ map information, it should 
be kept in mind that students at different grades 
; demonstrated these abilities with grade-wpropriate 
! materfeOs. 
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vSummary 

Ex^ining specific knowledge and skills demonstrated 
by indents at various points on the geography compos- 
ite scale reveals a variety of patterns in student perfor- 
mance. Generally, students across grades in the higher 
percentiles exhibited greater abilities to work with a 
range of geographic tools, create maps based on tabular 
or narrative data, grasp processes and relationships, 
bring outside knowledge to bear on answering questions, 
and analyze data. Perhaps predictably, these students 
were more likely to read more pages per day in school, 
spend more time on their homework, «md more fre- 
quently discuss their studies at home than their lower 
performing peers. (See Endnote 1.) 

When student performance on specific tasks is 
illustrated on item maps for each subscale, the percentiles 
for each subscale help to place these performances in 
perspective. Clearly, students found the assessment 
challenging. For each subscale across grades, a number 
of tasks fell at or above the 90th percentile. The NAEP 
1994 geography assessment was rigorous; many tasks 
demanded the ability to <tnalyze and interpret visual, 
tabular, and textual material. On the other hand (with 
some variation across grades and subscales), some tasks 
also fell at or just above the 50th percentiles, suggesting 
that half the student population was able to answer a 
reasonable portion of assessment questions. 

Patterns of student subgroup performance on 
the three geography content areas for the most part 
reflected those shown on the composite scale. For 
example, across the three content areas, a positive 
relationship existed between increasing levels of parental 
education and student scores, and students attending 
nonpublic school outperformed students attending 
public schools. In addition, with only a few exceptions, 
males outperformed females across content areas. 



Endnotes 

1. Results for pages read in school «md time spent on 
homework are based on collapsed data. The data do 
not necessarily imply a direct, positive linear 
relationship between student performance on the 
NAEP geography assessment and students’ reports of 
pages read in school and time spent on homework. 
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The NAEP 1994 geography assessment required students 
to analyze data in a variety of formats, explain complex 
geographic phenomena and processes, and show knowl- 
edge of relationships among human problems and 
events and geographic phenomena. A variety of stimuli, 
such as maps, photographs, tables, and charts, were used. 

In addition, a large percentage of assessment time was 
devoted to constructed-response questions, for which 
students had to write their own answers and create 
maps and tables. 

Many students found the assessment difficult. For 
the nation as a whole, approximately one-quarter 
reached the Proficient achievement level, representing 
solid academic performance. Furthermore, when 
student performance on specific tasks is illustrated on item 
• maps for each NAEP content area subscale, a number of 
tasks fell at and above the 90th percentiles for each 
subscale, indicating that only top-performing students 
could fully respond to these questions. On the other 
hand, approximately 70 percent of students were able to 
reach the Basic achievement level, which denotes partial 
mastery of knowledge and skills fundamental for profi- 
cient grade-level work. And (with some variation across 
grades and subscales), some tasks also fell near the 50th 
percentile on the subscale item maps, suggesting that 
half of the students sampled were able to answer a 
reasonable portion of assessment questions. 

Some of the patterns in performance are character- 
istic of patterns seen in other NAEP assessments, such 
as reading and U.S. history. For example, White and 
Asian students generally had higher average scale scores 



than did Black and Hispanic students. In addition, 
students whose parents had higher levels of education 
outperformed their peers who reported lower levels of 
parental education, and students in nonpublic schools 
outperformed those in public schools. On the other 
hand, while male students Outperformed female stu- 
dents only at grade 12 in the U.S. history assessment, 
male students outperformed female students at all grades 
in the NAEP 1994 geography assessment. 

Subscale performances for geography generally 
mirrored performances on the composite scale, with 
some interesting variations. For example, at grade 4 
males outperformed females on tasks assessing the 
geography content area Space and Place, while there 
were no significant performance differences in the other 
two content areas. 

Performance associated with home support and 
other background variables also was consistent with 
that found in other NAEP assessments. Frequent televi- 
sion watching was associated with lower scale scores, 
while literacy materials in the home and discussing 
studies at home were associated with hi^er scale scores. 

It is encouraging that exposure to geography at grade 8 
and to social studies at grade 4 were associated with 
higher average geography scores, and the more semes- 
ters of potentially geography-related course work done 
by twelfth graders, the better their performances. 

While the NAEP results presented in this report 
cannot be used to draw causal inferences, they do point 
out interesting characteristics and patterns of student 
performance. Future research and other projects and 
analyses can use NAEP data to shed more light on 
relationships between performance and background data, 
which in turn can be used by policymakers, educators, 
-and citizens to bring change to the United States 
educational system. - - 
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Overview of Procedures 
Used in the NAEP 1994 
Geography Assessment 

Introduction 

The conduct of a large-scale assessment of educational 
progress entails the successful coordination of a 
multitude of projects, committees, procedures, and 
tasks. This appendix provides an overview of the NAEP 
1994 geography assessment’s primary components: 
framework, development, administration, scoring, and 
analysis. A more extensive review of procedures and 
methods used in the assessment will be included 
in a subsequent technical report: The NAEP 1994 
Technical Report. 



The framework underlying the NAEP 1994 geography 
assessment reflects current consensus among educators 
and researchers about the study of geography. 

The framework’s purpose was to provide a view of 
geography on which to base the NAEP assessment 
Developing this framework and the specifications that 
guided development of the assessment involved the 
critical input of hundreds of individuals across the 
country, including representatives of national education 
or gan izations, teachers, parents, policymakers, business 
leaders, and the interested generd public. This 
consensus process was managed by the Council of Chief 
State School Officers for the National Assessment 
Governing Board. 

The framework sets forth a broad content matrbt 
that describes geography in terms of content areas and 
cognitive dimensions. Figure A.1 illustrates this content 
matrix. 
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The assessment framework specified the aspects of 
geography to be measured and the percentage of 
as4essment time that should be devoted to each. Thble 
A.1 presents the target percentage distributions of 
content areas as specified in the framework, along with 
the actual percentage distributions in the assessment 
The actual content of the assessment was consistent with 
the targeted distribution. 

The Assessment Design 

Each student in the assessment received an assessment 
booklet containing general background questions, 
geography questions, a set of background questions 
specific to social studies or geography, and a set of 
questions that determined students’ motivation and 
familiarity with the assessment tasks. The geography 
questions and their stimulus material were arranged 
into blocks. Students were given either two 25-minute 
blocks or one 50-minute block. At the fourth grade, only 
25-minute blocks were used. 

The grade 4 assessment consisted of six 25-minute 
blocks, two of which were also administered to eighth- 
grade students. Each block contained single questions 
and sets, a variety of stimulus material, and a combina- 
tion of multiple-choice and constructed-response 
questions. One or two of the constructed-response 
questions in each block required an extended response. 
A total of 59 multiple-choice questivons, 23 short con- 
structed-response questions, and 8 extended constructed- 
response questions were administered at grade 4. 



The grade 8 assessment consisted of six 25-minute 
blocks and one 50-minute block. The six 25-minute 
blocks include two also administered at grade 4 (total of 
31 questions) and two also administered at grade 12 
(total of 34 questions). The 25-minute blocks followed 
the same pattern as those for grade 4. The 50-minute 
block included questions all focused on a particular theme 
and contained two extended constructed-response 
questions. A total of 84 multiple-choice questions, 32 
short constructed-response questions, and 9 extended 
constructed-response questions were administered 
at grade 8. 

The grade 12 assessment consisted of sa 25-minute 
blocks (including two also administered at grade 8) 
and one 50-minute block. These blocks followed the 
same pattern as those for grades 4 and 8, except that 
all blocks (with the exception of one block for both 
grades 8 and 12 that included one extended constructed- 
response question) included two extended constructed- 
response questions. The 50-minute block included 
questions all focusing on a particular theme and 
contained two extended constructed-resp<Mise questions. 
A total of 85 multiple-choice questions, 25 short con- 
structed-response que'fions, and 13 extended constructed- 
response questions were administered at grade 12. 

The assessment design allowed for maximum 
coverage of the domain of geography at each grade, 
while minimizing the time burden for any ohe student. 
This was accomplished through the use of matrix 
sampling, in which a representative sample of students 
took each portion of the assessment Individual students 
were required to take only a small part; however, the 
aggregate results across the entire assessment allow for 
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broad reporting of geography abilities for the targeted 
population. 

• In addition to matrix sampling, the assessment 
design used a procedure for distributing booklets that 
controlled for position and balance effects. Students 
received different blocks of questions in their booklets 
according to a specific design. Balanced incomplete 
block (BIB) spiraling was used to assign blocks of 
questions in a manner that balanced the positioning 
of blocks across booklets and balanced the pairing of 
blocks within booklets. The spiraling aspect of this 
procedure cycles the booklets for administration so that 
typically only a few students in any assessment session 
receive the same booklet. 



Teacher and School Questionnaires 



One of the most important parts of NAEP’s efforts to 
document the nature of students’ achievement is the 
collection of contextual information regarding students’ 
school experiences. As a part of the 1994 geography 
assessment, NAEP administered a questionnaire to 
teachers responsible for teaching social studies or 
geography to students who participated in the fourth- or 
eighth-grade assessments. In addition, the principals or 
other administrators of sampled schools at all grades 
were asked to complete a school questionnaire. These 
questiormaires were developed in consultation with an 
expert panel. These instruments focused on five areas: 
instructional content, instructional practices and 
experiences, teacher characteristics, school conditions , 
and contexts, and conditions outside the school (i.e., ' 

home support, out-of-school activities, and attitudes). 



The fourth- and eighth-grade social studies iid 
geography teacher questionnaires were composed of two 
sections each. One section contained questions about 
teachers’ background, education, and resources. The 
other section contained questions about teachers* recent 
exposure to training in various areas of geography 



education, the structure and nature of their classroom 
instruction, and the types of materials and approaches 
they use in teaching geography. 

Because the sampling of teachers for the teacher 
questionnaires was based on participating students, the 
teachers’ questiormaire responses do not necessarily 
represent all fourth- and eighth-grade teachers 
in the nation. Rather, they represent teachers of the 
representative sample of students in the assessment 
Consequently, these findings portray, the nature of 
students’ instructional experiences and the background 
of their teachers. 

It is important to note that in this report, as in all 
NAEP reports, the student is the unit of analysis — 
even when information from teacher or school 
questionnaires is being reported. Using the student as 
the unit of analysis makes it possible to link students’ 
performance with their instructional and background 
experiences, thus providing a rich source of relevant 
information for educators and researchers. Although 
this approach may provide a different perspective from 
other studies that simply report information about 
teachers or schools, it is consistent with NAEP’s goal of 
providing information about the educational cont^ „ 
and performance of students. 

Some students selected for the assessment were 
judged by school authorities to be incapable of 
. meaningful participation in the assessment because 
had limited English-language proficiency, were mentally 
r. challenged, or were functionally disabled. (See the 
‘' Limited English Proficient and Individualised Education 
Plan section in this appendix.) For each student 
excluded from the assessment, schools were required to 
complete a questionnaire about the characteristics of 
that student and the reason for exclusion. < r . \ 
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NAEP Geography Samples 

The results presented in this report are based on 
nationally representative probability samples of fourth-, 
eighth-, and twelfth-grade students. The samples were 
selected using a complex multistage sampling design 
involving the sampling of students h'om selected schools 
within selected geogrzgphic areas across the country. The 
sample design had the following stages: 

1) selection of primary sampling units (PSUs) — 
geographic areas defined as counties or groups of 
counties); 

2) selection of schools (both public and nonpublic) 
within the selected areas; and 

3) selection of students within selected schools. 



Each selected school that participated in the 
assessment, and each student assessed, represents a 
portion of the population of interest To niake valid 
inferences from the student samples to the respective 
populations from which they were drawn, sampling 
weights are needed. Sampling weights account for 
disproportionate representation due to oversampling of 
nonpublic schools and of students attending schools 
with high concentrations of Black or Hispanic students 
or both. Lower sampling rates for very small schools 
must also be accounted for with the sampling weights. 

Table A.2 provides a summary of the weighted and 
unweighted student sample sizes for the geography 
assessment The numbers reported include both public 
and nonpublic school students. 
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^nd Individualized Education Plan 
! (lEP) Students 

I It is NAEP’s intent to assess all selected students. 

! Therefore, every effort is made to ensure that all 
I selected students who are capable of participating in the 
assessment are assessed. However, some students 
sampled for participation in NAEP can be excused from 
the sample according to carefully defined criteria. 
Specifically, some students identified as having Limited 
English Proficiency (LEP) or having an Individualized 
Education Plan (lEP) may be incapable of participating 
meaningfully in the assessment. TTiese students are 
identified as follows: 

j Students classified as LEP may be excluded fi’om the 

assessment if 

i > the student is a native speaker of a language other 
than English; AND 

► the student has been enrolled in an English- 
speaking school less than two years; AND 

^ the school staff most familiar with the student have 

judged the student to be incapable of taking part in 
the assessment. 



Scoring 

Materials from the 1994 assessment were shipped to 
National Computer Systems in Iowa City for processing. 
Receipt and quality control were managed through a 
sophisticated bar-coding and tracking system. After all 
appropriate materials were received from a school, they 
were forwarded to the professional scoring area where 
the responses to the constructed-response questions 
were evaluated by trained staff using guidelines 
prepared by NAEP. Each constructed-response question 
had a unique scoring guide that defined the criteria to 
be used in evaluating students’ responses. The extended- 
response questions were evaluated with four-level 
rubrics. Many of the short constructed-response 
questions were rated according to three-level rubrics 
that permitted partial credit to be given. 

For the NAEP 1994 geography assessment, 
approximately 375,000 student responses were scored. 
This figure includes a 25-percent rescore to monitor 
interrater reliability. Comparable to the NAEP 1994 
reading and U.S. history assessments, the overall 
percentages of agreement between scorers for the 1994 
geography reliability samples were 93 percent at grade 4, 
93 percent at grade 8, and 90 percent at grade 12. 



Students classified as lEP may be excluded if 

► the student is mainstreamed less than 50 percent of 
the time in academic subjects and is judged to be 
incapable of taking part in the assessment, OR 

► the lEP team has determined that the student is 
incapable of taking part meaningfully in the 
assessment. 

When there Is doubt, the student is included in 
the assessment. 



For each student excused from the assessment, 
school personnel complete a questionnaire about the 
characteristics of that student and the reason for 
exclusion. 






Data Collection 

The NAEP 1994 geography assessment was conducted 
from January through March 1994, with some make-up 
sessions in early April. As with all NAEP assessments, 
data collection for the 1994 assessment was conducted 
by trained field staff. For the geography assessment, this 
was accomplished by Westat, Inc., staff. 



Data Analysis and IRT . 

■ ■ ■■ ■ V'l . 

Subsequent to the professional scoring, all information 
was transcribed to the NAEP database at Educational 
Testing Service. Each processing actiyi^ conducted 
with rigorous quality control. After the jBS^ment 
information had been compiled m.the di^a^^ 
were weighted according to the popu&tfoii structure. 
The weighting for the samples reflectetf^ probability 
of selection for each student as a rci^tyt>fj^e sampling 
design, adjusted for nonresponse. stotification, 

the weighting ensured that the repr^ntalSbh of certain 
subpopulations corresponded to figure from the U.S. 
Bureau of the Census and the Current Population 
Survey.* ^ , . rv 

Analyses were then conducted to determine the 
percentages of students who gave various responses to 
each geography and background question. In 
determining these percentages for the geography 
questions, a distinction was made between missing 
responses at the end of a block (i.e., missing responses 
subsequent to the last question the student answered) 
and missing responses prior to the last observed 
response. Missing responses before the last observed 
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response were considered intentional omissions. 
Mining responses at the end of the block were 
coraidered “not reached” and treated as if the questions 
had not been presented to the student In calculating 
response percentages for each question, only students 
classified as having been presented the question were 
included in the denominator of the statistic. 

It is standard ETS practice to treat all nonrespondents 
to the last question in a block as if they had not reached 
the question. For multiple-choice and short constructed- 
response questions, this practice produces a reasonable 
pattern of results in that the proportion reaching the 
last question is not dramatically smaller than the 
proportion reaching the next-to-last question. However, 
for blocks that ended with extended constructed- 
response questions, the standard ETS practice would 
result in extremely large drops in the proportion of 
students attempting the final question. A drop of such 
magnitude seemed somewhat implausible. Therefore, 
for blocks ending with an extended constructed-response 
question, students who answered the next-to-last question 
but did not respond to the extended constructed- 
response question were classified as having intentionally 
omitted the last question. 

Item response theory (IRT) was used to estimate 
average geography scale scores for the nation and for 
various subgroups of interest within the nation. IRT 
models the probability of answering a question in a 
cerWn way as a mathematical function of skill. The 
niam purpose of IRT analysis is to provide a common 
scale on which performance can be compared across ^ 
groups such as those defined by grades and characteristic^ 
including race/ethnicity and gender. 

Because of the BIB-spiraling design used by NAER 
students do not receive enough questions about a ‘ V!': 
spedfic topic to provide reliable information about ' \ \ 
iridividual performance. Traditional test scores for ^ 
individual students, even those based on IRT, would Iwd 
to misleading estimates of population characteristics, 
such as subgroup means and percentages of students ait 
or above a certain achievement level. Consequently, 
NAEP constructs sets of plausible values designed to " ' 
represent the distribution of scores in the population. 

A plausible value for an individual is not a scale score for 
that individual but may be regarded as a representative 
value from the distribution of potential scale scores for 



dl studente in the population with similar character- 
istics and identical patterns of item response. Statistics 
describing performance on the NAEP geography scale 
are based on the plausible values. They estimate values 
that would have been obtained had individual 
performances been observed — that is, had each student 
responded to a sufficient number of cognitive questions 
so that performance could be precisely estimated.* 

For the NAEP 1994 geography assessment, within- 
grade scales were created to report performance for 
each subscale (i.e., content area). Similar scaling 
procedures were used to establish each of the three 
subscales. Specifically, three within-grade subscales 
(one for each grade) were established for each of the 
geography content areas. The within-grade subscales for 
grades 4 and 12 were then linked to the grade 8 subscale 
to form a common reporting metric. This common 
reporting metric, which runs from 0 to 500 for each of 
the subscales, was established so that the mean scores 
across all three grades is 250 and the standard deviation 
of the scores is 50. 

The composite NAEP geography scale was produced 
as a weighted average of the subscales, the weights 
being given by the target percentages shown in Thble A.1. 
The reporting metric of the composite scale, which also 
runs from 0 to 500, was again established so that the 
mean score across all three grades is 250. No constraints 
were imposed on the standard deviation of tfie cross- 
grade composite scores. • 

It may be helpfu] here to provide sonte guidance to 
the reader of this report about the of q^-grade 

and cross-scale inferences that are apprdpnate. ITe use 
of a common cross-grade metric for the subscales and 
the composite scale was motivated primarily tiy issues 
of convenience in the reporting of results. In producing 
the subscales, IRT pai^eters for qui^bniToommon 
across the grades were not consti^ned to equal. As 
a result of this scaling convention, cros^grade com-' 
parisons of scale score averages, both at the subscale 
level as well as for the composite scale, nay not be 
meaningful. Similarly, scale score differences (e.g., 
between subscale or composite scale averages for males 
and females) should probably not be compared across 
grades. The reader is best served by focusing on within- 
grade group comparisons and inferences. . , 
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The use of a common scaling procedure for each 
of the subscales does provide for some within-grade 
normative meanings across subscales. For example, at 
grade 4, a score of 200 is one cross-grade standard 
deviation unit below the cross-grade average for each of 
the geography subscales. Similarly, a score of 210 is 
four-fifths of a cross-grade standard deviation below the 
cross-grade average for each of the scales. A group, 
for example males, scoring 200 on the Space and Place 
subscale and 210 on the Environment and Society 
subscale did indeed perform “better” on the latter than 
on the former in a cross-grade normative sense. 

However, other inferences about relative performance 
in, say, a percent-correct metric do not necessarily 
follow. For example, a score of 200 on the Space and 
Place subscale may imply a higher expected percent- 
correct score on the collection of assessment exercises 
that define that scale than is implied by a score of 210 
on Environment and Society subscale. Thus, 
continuing with the current example, performance on 
the Space and Place subscale was better in the percent- 
correct sense than performance on the Environment 
and Society subscale. 

In Chapter 5, performance across subscales is 
compared 1^ examining patterns of subgroup differences 
(i.e., patterns of statistic^ significance between subgroups 
across the three subscales). These patterns are discussed 
separately for each of the three grades. Within-grade 
inferences based on such comparisons are defensible 
given the limited degree of comparability that exists in 
the subscale reporting metrics. As noted above, other 
types of inferences (e.g., inferences involving subscale 
score difterences) may be less defensible. 

The subscales summarize student performance 
across all three question types in the assessment 
7 (multiple-choice, short constructed-response, and 
^ extended constructed-response). Iii producing these 
subscales, two IRT models were used. Multiple-choice 
questions were scaled using the three-parameter logistic 
(3PL) model; and short constructed-response questions 
rated according to a three-level rubric, as well as 
extended constructed-response questions rated on a 
four-level rubric, were scaled using a generalized 
partial-credit (GPC) model.® Developed by ETS and first 
used in 1992, the GPC model permits the scaling of 
questions scored according to multipoint rating 
schemes. The model takes full advantage of the 
•information available from each of the student response 
categories used for these more complex constructed- 
response questions. 



The geography scale is composed of two types of 
questions: multiple-choice and constructed-response 
(scored according to a partial-credit model). One natural 
question about the scale concerns the amount of 
information contributed by each type of question. 
Unfortunately, this question has no silfiple answer 
for the NAEP geography assessment, due to the complex 
procedures used to form the composite geography scale. 

The information provided by a given question is 
determined by the IRT model used to scale the question 
and is a function of proficiencies.* Thus, the answer to 
the query “How much information do the different types 
of questions provide?” will differ for each level of 
geography proficiency. When considering the composite 
geography scale, the answer is even more complicated. 
The geography data are sealed separately by the 
geography content areas. As discussed on ^e previous 
page, the composite scale is a weighted combination of 
these subscales. IRT information functions are only 
strictly comparable when they are linked on a common 
scale. Because the composite scale is based on three 
separate calibrations, without any common item linking, 
there is no direct way to compare the information 
provided by the questions on the composite scale. 



NAEP Reporting Groups 

Findings from the NAEP 1994 geography assessment are 
presented for groups of students defined by shared , 
characteristics. Data are reported for sub^oups only 
where sufficient numbers of students and adequate 
school representation are present There must be at 
least 62 students in a particular subgroup, and these 
students must come ftom at least sue different PSUs (see 
-description of sampling design on page 84). Data for all 
students, regardless of whether their subgroup was 
reported separately, were included in computing overall 
national and regional results. 

The reporting subgroups presented in this report 
include: region, race/ethnicity, gender, parents' highest 
level of education, type of school, and type of location. 
Definitions of these subgroups are provided on the 
following page. 
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Region. Results are reported for four regions of the 
nation: Northeast, Southeast, Central and West States 
induded in each region are shown in Figure k2. All 
50 states and the District of Columbia are listed. U.S. 
territories were not assigned to a region. 

Race/Etimicity. The race/ethnicity variable is an 
imputed dehnition of race/ethnicity, derived from up to 
three sources of information. This variable is used for 
race/ethnicity subgroup comparisons. TWo questions 
from the student demographics questionnaire were used 
in the determination of derived race/ethnicity: 



If you are Hispanic, what is your Hispanic 
background? 

O I am not Hispanic. 

O Mexican, Mexican American, or Chicano 
O Puerto Rican 
O Cuban 

O Other Spanish or Hispanic background 



L 



Students who responded to this question by filling 
in the second, third, fourth, or fifth oval were 
considered Hispanic. For students who filled in the first 
oval, did not respond to the question, or provided 
information that wais illegible or could not be classified, 
responses to the following question were examined in an 
effort to determine race/ethnidtyl '* ' ' ’ ' 

■- • ' .-rnmwr... i-: ■■ 
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Which best describes you? 

O White (not Hispanic) 

O Black (not Hispanic) 

O Hispanic (“Hispanic” means someone who is 
Mexican, Mexican American, Chicano, 

Puerto Rican, Cuban, or other Spanish or 
Hispanic background.) 

O Asian (“Asian” means someone who is Chinese, 
J^)anese, Korean, Vietnamese, or other Asian 
background.) 

O Pacific Islander (“Pacific Islander” means 
someone who is from a Filipino, Hawaiian, or 
other Pacific Island background.) 

O American Indian or Alaskan Native (“American 
Indian or Alaskan Native” means someone who is 
fi'om one of the American Indian tribes, or one of 
the original people of Alaska.) 

O Other 



Students’ race/ethnicity was then assigned to cor- 
respond with their selection. For students who filled in the 
seventh oval (“Other”), provided illegible information or 
information that could not be classified, or did not 
respond at all, race/et^city as provided firpm school . 
records was used. ' ' ‘ ' » '‘’'' T' 

Derh^ race/etlmicity could not be determined 
for students who did not respond to either of the i ’ 

demographic questions ^d for whom race/ett^dty was 
not provided by the school. * » . 
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Gender* Results are reported separately for males 
and females. 






i Parents* Highest Level Education Level. The parents’ 
education level variable is derived from responses to two 
questions in the student demographic questionnaire. 
Students were asked to indicate the extent of their 
mother’s education (How far in school did your mother 
go?) by choosing one of the following: 

O She did not finish high school. 

O She gradiwted from high school. 

O She had some education after high school. 

0 She graduated from college. 

0 1 don’t know. 

Students were asked to provide the same information 
about the extent of their father’s education (How far in 
school did your fether go?) by choosing one of the 
following: 

O He did not finish high school. 

O He graduated from high school. 

O He had some education after high school. 

0 He graduated from college. 

0 1 don’t know. 

The information was combined into one parental 
education reporting category as follows: if a student 
indicated the extent of education for only one parent, 
that level was included in the data. If a student indicated 
the extent of education for both parents, the higher of 
the two levels was included in the data. For students 
who did not know the level of education for both parents 
or did not know the level of education for one parent 
and did not respond for the other, the parental 
education level was classified as unknown. If the student 

did not respond for both parents, the student was 

recorded as having provided no response. 

It should be noted that approximately one-third of 
fourth graders and one-tenth of eighth graders reported 
not knowing the education level of either of their 
parents. The percentages of students who reported not 
knowing their parents’ education level were larger for 
fourth-grade Hisi^ic students and for eighth-grade 
Black and Hispanic students compared to their White 
counterparts. (See 'ftd)leA.3) 

In addition, evidence from other surveys by the 
National Center for Education Statistics that gather data 
from students and parents indicates larger discrepancies 
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between students’ and parents’ reports for Black and 
Hispanic students as compared to White students. 

These differences between racial/ethnic groups are 
most evident at grade 8. As shown in Thble A.4, the 
correlations between students’ and parents’ reports of 
parental education were lower for Black and Hispanic 
students than for White students at both grades 8 and 12, 
although all correlations were higher in twelfth grade. 
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T)n>e of School. Results are reported by the type of 
school that the student attends: public or nonpublic. 
Nonpublic schools include Catholic and other nonpublic 
schools. Bureau of Indian Affairs (BIA) schools and 
domestic Department of Defense (DoD) schools are not 
included in either the public or nonpublic categories 
but are included in the overall national results. 
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Type of Location. Results are reported for students 
attending schools in three mutually exclusive location 
tyi^: central city, urban fringe/large town, and rural/ 
small town: 

Central City: This category includes central cities of 
all Standard Metropolitan Statistical Areas 
(SMSAs).® Central City is a geographic term and is 
not synonymous with “inner city.” 

Urban FringelLarge Towrv The urban fringe category 
includes all densely settled places and areas within 
SMSAs that are classified as urban by the U.S. Bureau 
of the Census. A Large Town is defined as a place 
outside a SMSA with a population greater than or 
equal to 25,000. 

RuraUSmall Tbum: Rural includes all places and 
areas with populations of less than 2,500 that are 
classified as rural by the U.S. Bureau of the Census. 
A Small Town is dehned as a place outside a SMSA 
with a population of less than 25,000 but greater 
than or equal to 2,500. 

As described earlier, the NAEP geography scale 
makes it possible to examine relationships between 
students’ performance and a variety of background 
factors measured by NAER However, the fact that a 
relationship exists behveen achievement and another 
variable does not r^c|^ the underlying cause of the 
relationship, which may be influenced by a number of 
other variables. Similitfk ^e ^sessment does not 
capture the influenicc of uruheasured variables. The 
results are most u^fril when they are considered in 
combination with other knowledge about the student 
population and the educational system, such as trends 
in instruction, changes in the school-age population, 
and societal demands and expectations. 



Estimating Variability 

Because the statistics presented in this report are 
estimates of group and subgroup performance based on 
samples of students, rather than the values that could be 
calculated if every student in the nation answered every 
question, it is important to account for the degree of 
uncertainty associated with the estimates. 1\vo components 



of uncertainty are accounted for in the variability of 
statistics based on scale scores: 1) the uncertainty due to 
sampling only a relatively small number of students, 

^d 2) the uncertainty due to sampling only a relatively 
small number of questions. The variability associated 
with the estimated percentages of students with certain 
background characteristics or who answered a certain 
cognitive question correctly is accounted for by the first 
component alone. ' 

In addition to providing estimates of percentages of 
students and their average scores, this report provides 
information about the uncertainty of each statistic. 
Because NAEP uses complex sampling procedures, 
conventional formulas for estimating sampling 
variability that assume simple random sampling are 
inappropriate. NAEP uses a jackknife replication 
procedure to estimate stan^rd errors. Tbe jackknife 
standard error provides a reasonable measure of 
imcertainty for any information about students that can 
be observed without error. However, each student 
typically responds to so few questions within any 
content area that the score measurement for «my single 
student would be imprecise. In this case, using plausible- 
values technology makes it possible to describe the 
performance of groups and subgroups of students, but 
the underlying imprecision that makes this step 
necessary adds an additional component of variability to 
statistics based on NAEP scale scores.* , l v 

The reader is reminded that, like findings from all 
surveys, NAEP results are. also subject to other J^ds of 
errors including the effects of imperfect adjw^i^ent for 
student and school nonresponse, and other ifrinuiwa^ 
effects associated with the particular, instrumentation 
, and data collection methods. Nonsampling j0r|pr$ can be 
attributed to a number of sources: inability tn obtain I v 
complete information about all selected schopis in the , , 
sample (some students or schools refused to. psfrticipate,. 

‘ or students participated but answered onfy cprl^ .i .:,. * 
questions); ambiguous definitions; differences in inter- 
preting questions; inability or unwillingness, to give , , 

correct information; mistakes in recording, cbding, or 
scoring da^ and other errors of collectii^, prcK^ing, 
sampling, and estimating missing datau 'i^e o^t of 
nonsampling error is difficult to estimate! By tiieir ' ' 

nature, the impact of such errors cannot be refiected in 
the data-based estimates of uncertainty provided in ’ 
NAEP reports. ' ‘ it’! \ 
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Drawing Inferences from the Results 

The use oi confidence intervals, based on the standard 
errors, provides a way to make inferences about the 
population means and percentages in a manner that 
^tim^ps'^ uncertainty associated with the sample 

An estimated sample mean ± 2 standard errors 
represents an approximate 95-percent confidence 

ml! c ful /^o^'^ponding population quantity. This 
means that with approximately 95-percent certainty, the 

population of interest 

IS within ± 2 standard errors of the sample mean. 

As an example, suppose that the average score of 
student in a particular group was 256, with a standard 
error of 1.2. A 95-percent confidence interval for the 
population quantity would be as follows: 

Mean ± 2 standard errors 
256 ± 2 X 1.2 

256 ±2.4 

253.6, 258.4 

fh conclude with 95-percent certainty 

mat the averse scale score for the entire population of 
students in that group is between 253.6 and 258.4. 

Similar confidence intervals can be constructed for 
percent^es, provided that the percentages are not 
ertremely large (greater than 90) or extremely small 
(1^ thM 10). For extreme percentages, confidence 
intervals constructed in the above manner naay not be 

procedures for obtaining accurate 
confidence intervals are quite complicated. Thus 
comparisons involving extreme percentages should be 
interpreted with this in mind. 

To determine whether there is a real difference 

percentage of a certain 

attribute) for two groups in the population, one needs 
to obtain an estimate of the degree of uncertainty 
associated with the difference between the means or 
percent^es of these groups for the sample. When 
comparing two independent estimates, this estimate of 
the degree of uncertainty - called the standard error of 
the difference between the groups - is obtained by 



taking the square of each group’s standard error 
summing these squared standard errors, and then 
taking the SQuare root of this sum* 
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In a manner similar to that in which the standard 
error for an individual group mean or percentage is 
used, thP s^^dard error of the difference can^ used t 
help determine whether differences between groups in 
the population are real. The difference between the 
mean sc^e score or percentage of the two groups ± 2 
tandard errors of the difference represents an 
approHmate 95-percent confidence interval. If the 
resulting interval includes zero, there is insufficient 
ewdence to claim a real difference between groups in 
populatton. If the interval does not contain zero, th 
difference be^een groups is statistically significant 
(different) at the .05 level. 

T^e procedures described in this section, and the 

‘P a 95:percent confi- 

dence interval) are based on statistical theory that 

^umes that only one confidence interval or test of 
statisttcal sigmficance is being performed. When one 

intervals, statistical theory 
mdiwtes that the certainty associated with the entire « 
of mter^^ is less than that attributable to each mdividu^ 
comparison from the set If one wants to hold the 
wrtanty Iwel for a specific set of bihparisons at a 
^rticular level (e.g., 95 percent), adjustments (called 
multiple-comparisons procedures) need to be made A 
mwe complete discuMion of the multiple-comparisons 
procedures is presented in theMAffP/Sfid Technical 
Report. , 

The s^dard errors for means and percentages 

retried by NAEP are statistics antfare-Sbject^^ * 

cases, typically 

^®*^ number of 

stedents (or when the group of students is enrolled in a 
small number of schools), the amount of uncertainty 
^sociated wi^ the standard errors may be quite large 
Throu^out this report, estimates of standard errors^ ' 
subject to a large degree of uncertainty are desi^d 
by the symbol “1”. In such cases, the standard eSrs — 

®"®® or significance tests involving 

these standard errors - should be interpreted cautiouslT 
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Describing Students’ 
Geograplo? Perfonnance 

This appendix contains detailed information about 
the procedures used for describing students’ geography 
knowledge and abilities and profiling students’ study 
habits. Chapter 5 presents the results of these 
procedures. 

Performance Descriptions Based on 
the Geography Composite Scale 

A procedure known as scale anchoring was used to 
develop descriptions of student performance at selected 
points on the NAEP geography composite scale. The 
scale points selected for anchoring reflect three levels of 
geography knowledge and abilities corresponding to 
lower, middle, and higher performing students. These 
levels correspond to the 25th, 50th, and 90th percentile 
points on the composite scale as established by the 
performance of students in 1994 — the first assessment 
administered under NAEP’s current geography 
firamework. 

Around each percentile point, a band was built to 
define a range of scale scores. Students described as 
being at a pa^icular level were within a range of five 
percentile points on either side of the specified scale 
point For example, the 50th percentile was defined as 
the region between the 45th and 55th percentile points 
on the scale. A question was identified as anchoring 
at a percentile point on the scale if at .,t 65 percent of 
the students within that percentile band answered the 
question successfully. (The criterion was set at 74 
percent for multiple-choice questions to correct for the 
possibility of answering correctly by guessing.) 

After defining the bands of the scale to be anchored, 
the next step in the process was to identify: (1) questions 
answered correctly for dichotomously scored questions 
and (2) questions answered at a particular score level for 
partial credit constructed-response questions. Because 
the extended constructed-response questions were 
scored according to four levels of performance, each 
extended constructed-response question was treated as 
three distinct questions corresponding to scores of 



“Partial or better,” “Essential or bet'.er,” and “Complete 
These distinct score levels were then analyzed in the 
same manner as questions scored dichotomously, as 
either correct or incorrect. Thus, for example, an 
extended constructed-response question might anchor 
at the 50th percentile for “Partial or better” responses 
and at the 90th percentile for “Essential or better” 
responses. 

A committee of geography education experts, 
including teachers for the grades involved, college 
professors, state curriculum supervisors, and researchers, 
was assembled to review the sets of questions identifiec 
for each percentile band. The committee was divided 
into three groups, one for each grade. Each group 
examined and analyzed questions that anchored at the 
25th, 50th, and 90th percentiles to determine the 
specific geography knowledge and abilities associated 
with each question. 

Committee members were also given the sets of 
quMtions at each grade that “did not anchor” to inform 
their decisions about what students could do by seeing 
examples of what they could not do. Drawing on their 
knowledge of geography, committee members were 
asked to summarize student perfonnance by describing 
the knowledge, skills, and abilities demonstrated by 
students in rach of the score bands. 

The performance descriptions arc cumulative (that 
is, the abilities described for the lo\wi^i»rforming ' ’ 
students are considered to be amor^g&e abilities of 
students performing at higher points; on the scale). 
Therefore, the foil description of sfodejits’ geography 
joiowledge and abilities in the middic.^e band would 
include those abilities described at the lower band. 
Similarly, the abilities of students perfoihnihg at the 
_ higher scale band include the geognqdiy abilities 
- described for students at the middle^aibwer bands. 

Profiling Students’ Study Habits 

Using the scale bands defined for the anchoring process 
described above, the profiling of students’ study habits 
was accomplished by examining the responses of 
students within those bands to selected background 
questions. Ibbles B.l, B.2, and B3 contain a complete 
presentation of students’ responses to the three 
background questions highlighted in Chapter 5. The 
percentages that appear in the tables are conditional 
on the anchor scale point That is, they are percentages 
of students who scored within a five pe - centile point 
range on either side of the specified scale point 
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Rtsponses of Students Neor Selected Percentile Points 
to Generoi Study Hobit Questions 
Grode 8 
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Item-Mapping Procedures 

* To map questions to particular points on the NAEP 
geography subscales, a response probability convention 
had to be adopted that would divide those who had a 
higher probability of success from those who had a 
lower probability. Establishing a response probability 
convention has an impact on the mapping of assessment 
questions onto the geography subscales. A lower 
boundary convention maps the geography questions at 
lower points along the geography scales, and a higher 
boundary convention maps the same questions at higher 
points along the scales. The underlying distribution of 
geography skills in the population does not change, but 
the choice of a response probability convention does have 
an iinpact on the proportion of the student population 
that is reported as "able to do" the questions on the 
geography scales. 

No point along the probability scale is clearly 
superior to any other point. If the convention were set 
with a boundary at 50 percent, those above the 
boundary would be more likely to get a question right 
than to get it wrong, while those below that boundary 
would be more likely to get the question wrong than 
right While this convention has some intuitive appeal, 
it was rejected on the grounds that having a 50/50 
chance of getting the question right shows an 
imufficient degree of mastery. If the convention were set 
with a boundary at 80 percent, students above the 
criterion would have a high probability of success with a 
question. However, many of the students below this 
criterion show some level of geography ability that such 
a stringent criterion would ignore. In particular, those 
in the range between 50 and 80 percent correct would 
be more likely to get the question right than wrong, yet 
would not be in the group described as “able to do” the 
question. 

In a compromise between the 50-percent and the 
80-percent conventions, NAEP has adopted two related 
response probability conventions: (1) 74 percent for 
multiple-choice questions (to correct for the possibility 
of answering correctly by guessing); and (2) 65 percent 
for constructed-response questions (where guessing is 
not a factor). These probability conventions were estab- 
lished, in part, based on an intuitive judgment that they 
would provide the best picture of students’ geography 
knowledge and skills. 






Some additional support for the dual conventions 
adopted by NAEP was provided by Huynh (1994).» He 
examined the IRT information provided by questions, 
according to the IRT model used in scaling NAEP 
questions. (“Information” is used here in a technical 
sense. See the NAEP 1994 Technical Report for details.) 
Following Bock (1972),* Huynh decomposed the item 
information into that provided by a correct response 
[P (6) *I (6)] and that provided by an incorrect response 
I(l-P («)) *! (e)]. Huynh showed that the item informa- 
tion provided by a correct response to a constructed- 
response question is maximized at the point along the 
geography scale at which two-thirds of the students get 
toe question correct (for multiple-choice questions, 
information is maximized at toe point at which 74 ’ 
percent get the question correct). However, maximizing 
toe item information I («), rather than the information 
provided by a correct response [P («) *I (fi)], would 
imply an item mapping criterion closer to 50 percent. 



Endnotes 

1. Huynh, H. (1994, October). Some technical aspects 
of standard setting. Paper presented at toe Joint 
Conference on Standard Setting for Large-Scale 
Assessment, Washington, DC. 

2. Bock, R. D. (1972). "Estimating item parameters and 
latent ability when responses are scored in two or 
more latent categories.” />syc/iome/ra:a, 37, 29-51. 
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appendix c 



Sample Questions 
from the NAEP 1994 
Geogtaphy Assessment 

This appendix presents additional sample questions and 
student responses selected for each grade to exemplify 
the range of exercises included in the NAEP 1994 
geography assessment (A different set of sample 
questions and student responses are presented in 
Chapter 1, and entire sample blocks of questions can 
be found in the NAEP 1994 Geography: A First Look 
report, pages 27 to 67.) For each question, the 
geography content area being addressed is indicated. 



For multiple-choice questions, the correct answer is 
marked (►). For constructed-response questions, an 
abbreviated scoring rubic is provided. The sample 
student responses have been reproduced from 
assessment booklets and represent typical student 
performance. 

The table accompanying each sample question 
presents two types of percentages: (1) the overall per- 
centage of students within a grade who answered the 
question successfully and (2) the percentages of stu- 
dents within each of the achievement level intervals — 
Bosic, Proficient, and Advanced — who answered the 
question successfully. For grades 4 and 12, the percent 
^es for students within the Advanced achievement lev- 
interval are not presented, however, because of the sma 
sample sizes. The table also includes the percentages of 
students below Basic who successfully answered the 
questions. (Sample size criteria for reporting results an 
described in Appendix A.) 
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What would be the best title for this picture? 
A The Tides 
►B The Water Cycle 
C The Seasons 
D Ocean Currents 




Geography Content Area: Environment and Society 
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Describe two important effects that a nu^or oil spill in an ocean can have on the 
environment or on people’s lives. 



I 

I 

) 

I 



Geography Content Area: Environment and Society 



Gradb4 
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Cowpleta 
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AdgevMMrt Uval faf arvob 
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^ aLa ^ ^ li La ^IL AC ALa# ^ 

iiM sonoora tiiois vi nii piriMiDycs ippiv ■ pViiM nnii n im wnn 7>pifTiwuiMni)f mcn, tot •oqi 

pipuiAlA aI iriKftff lIlA mIm flf MA vInIa pA^ulAliAAil vMlii piiS Af MiHAllWAStMrfwi AfTIfS aI Aa ASARMAA for Aa SAAIpfo. 



Sample Response (Score of 3): 



Describe two important effects that a nudor oU spill in an ocean can have on the 
environment or on people’s lives. 



! 

i 



Oi l h\ c^pfl j^ up fl ivl tfiem 

i.rry. f At^r\r^ gWltli j j Jff A O , — : 



A Complete response (score of 3) describes two effects of an oil spill in a distant place. Effects 
described may be environmental (for example, pollutes beaches, pollutes air, kills living 
organisms), economic (for example, ruins fishing industry, ruins tourist industry), or political 
(for example, causes disputes over who <$ responsible for cleanup). 



I 

I 
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The map shows tiut one part of the country has more major highways than the 
other part of the country. Why is this? 

►a There are more people and cities in the eastern part of the country. 

B It is easier to build highways in the eastern part of the country. 

C Cars are not an important form of transportation in the western part of the country. 
D States are larger in the western part of the country. 



Geography Content Area; Spatial Dynamics and Connections 
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I 1) In tile box below, draw a map of an island, 

i 2) On the bland, put in the following details: 

I 

j i • Mountains along the west coast 

i • A lake in the north 

! • Houses along the east coast 

j • Forests in the south 

I ^ sure to use the symbols shown in the key below. 

Use your colored pencib to help you draw the map. 



1 



! 



I 

j- 

i 

































































































































































Key: 

Mountains Forests 

‘^ke . f 
666 Houses , . *'* -.1 ** 




Geognq;)!^ Content Area: Space and Place 
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piSiiblwrflitirW,t><di»hfiaiibhs»siAii i wa»illia|feyniinwlwDileiibtjirrw«ISwi i ll»i a hrlb»«[i>h. 
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Sample Response (Score of 3): 











o 
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Key; 

Mountains 

Co y 

Houses I ‘ I 



Forests 



An Essential response (score of 3) correctly indicates three or four of the elements. 
Simple Response (Score of 4): 




A Complete response (score of 4) correctly indicates all the elements. 
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Scunpie Questions and 
^Stiideift: Responses 



GRADE 8 



If you wanted to find out which page in the atlas had a map that showed the city 
of Dakar, where would you look? 

A The World Map Projections pages 
►B The Index 
C The World Facts page 
D The Earth Notes page 

Geography Content Area: Space <ind Place 



GndiS 


PsrcMtoge Cenect WHta 
AiUavMMBt Uvsl tatsrvob 


Overal PsrcMtoge 
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BthwBtak 
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97 ( 1 . 2 ) 
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Latitude on this map is represented by 
►A circles 
B shaded areas 
C straight lines 
D convergent lines 

Geography Content Area: Space and Place 
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if IMw, M fMPI 1w M VMM popiMnOi H WIM pwi W IIMB IVi SMinra wren || M 



utld 






People from many different countries live in New Yoi^ C%. Children speaking 
many different languages attend its public schools. This is main^ because New 
York City 

A has an efHcient transportation system 
B has a higher wage rate than other United States cities 
►C is a port of entry for people from other parts of the world 
D is the site of the United Nations headquarters 

Geography Content Area: Spacial Dynamics and Connections 
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Look at the oiap above, which shows three possible routes for a railroad line 
that will be built to connect Red City with Bluetown. 

Which route would be the least expensive to construct? 



Give two reasons why the route you chose would be the least esq^ensive. 



2 



Geography Content Area: Spacial Dynamics and Connections 
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OvmI BifCMrto||t 
CiMplito 


PifctitflQt ^Cbnolift' WHUi 
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Sanq>le Response (Score of 3): 



.y ^ V, 



Look at the map above, which shows three possible routes for a railroad line that 
will be built to connect Red City with Bluetown. 

Which route would be the least expensive to consmict? 



Ci^two reasons why the route you chose would be the least expensive. 

l.^SpUlfL b 3v^vKviL hji Tw. 

lAiAtid riC> rsiULL- -iko . 



r 



X 



JUL>C 






A Complete response (score of 3) indicates that C is the least expensive route to construct It 
gives two reasons why that may relate to A and B. 
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After we anchored our ships in the ocean and went ashore to explore, we nuurched 
west The forest was so ^ck we could only travel three miles in the first two 
days. Then we came to the mountains and climbed to the top. A rushing river 
flowed west out of the mountains. We continued to march two miles west and 
came down out of the mountains. IWo miles further we came to the coast It 
was obvious that the area we were exploring was an isthmus. 



In the box below, draw a map of the region described above. Be sure to include 
all of the geognythical elements mentioned in the description. Include a scale 
to indicate distances. 




Geography Content Area: Space and Place 



■ /iV'J 

Ovild Piiwrti||s 
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- 

% • •/ . 



Sample Response (Scove of 3): 







I- 



Thp'lpc!!.^ response (score of 3) includes a map in which three elements are correctly placed 
The response may be a peninsula or an island. ^ ^ 



may be a peninsula or an island. 

Sample Response (Score of 4); 





H 

• 1 1 



includes an accurate map in which at least four elements are 
iSS.*“‘^' river cor^^ 
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0 * 60 * 120 * 180 * 120 * 60 * 0 * 




Between 1961 and 1967, the area that had the most earthquakes was the 
A Mediterranean basin 
B mid-Atlantic Ocean 
C Caribbean Sea 

►D Pacific Ocean rim v ^ 

Geognq)hy Content Area: Space and Place 
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Environmental issues are viewed differently by people in different circumstances. 
Ebqilain how the artist makes this point in fte cartoon. 



Gcograplv Content Area: Environment and Society 
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Sample Response (Score of 3)t 



Environmental issues are viewed differently by people in different 
circumsunccs. Explain how the artist makes this point m the caitoon. 





Tf e: t 

rnan 


4.; )rci n\'her mc/n 


no f- 


"“fhn/) n --/K 


K — 


\r\ f\r\ ' n (ih\ mdh/ / On d 
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tin — ^ — 
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A Complete response (score of 3) mentions two different views (developed versus developing) 
and refers to trees and car pollution. An appreciation of tensioji may or may not be present 
Or, the response implies or states the hypocrisy that exists and^alks about the tree or the car. 



Sample Response (Score of 4): 



Environmenul issues are viewed differently by people in different 
circumsunces. Explain how the artist makes this point in^h^^i^OT 

























f ' Atf 

fluUw 



hlparrr-f^ 



An Este n Hal response (score of 4) discusses the environmental issues, tension (implied or 
stated between the two worlds), hypocrisy (not absolutely necessary if temion is clearly 
discussed), and two different viewpoints (developed versus developing). The discussion must 
be at the national level. 
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Which of the following countries has the largest volume and value of trade with 
the United States? 

A Japan 

B Great Britain 

Canada 

D Germany 



Geograpl^ Content Area: Spacial Dynamics and Connections 
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AVERAGE ANNUAL PRECIPITATION FOR LAKESIDE 



Season 


Average Inches 


Percent of TVital 


Spring 


5.0 


25 


Summer 


7.0 


35 


FaU 


4.0 


20 


Winter 


4.0 


20 


Total 


20.0 


100 



U*e the information in the table above to construct a pie chart on the figure 

beloWa Be sure to label all infoimation« You may use your ruler to draw the 
chart 




Geography Content Area: Space and Place 
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i 

Sample Response (Score of 3): 

* Use the information in the table above to construct a pie chart on the figure 

* below. Be sure to label all information. You may use your ruler to draw the 
chart 




A Complete response (score of 3) correctly charts the percentage of rainfall of the four seasons 
on the circle and correctly labels the segments (the minimum correct labels are the four 
seasons). 

' r, 
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The NAEP 1994 geography assessment was a 
collaborative effort among staff from the National 
Center for Education Statistics (NCES), the National 
Assessment Governing Board (NAGB), Educational 
Testing Service (ETS), Westat, and National Computer 
Systems (NCS). The program benefited from the 
contributions of hundreds of individuals at the state and 
local levels — governors, chief state school officers, 
state and district test directors, state coordinators, and 
district administrators — who tirelessly provided their 
wisdom, experience, and hard work. Most importantly, 
NAEP is grateful to students and school staff who made 
the assessment possible. 

The assessment was funded through NCES, in the 
Office of Educational Research and Improvement of 
the U.S. Department of Education. The NCES staff — 
particularly Jeanne Griffith, Gary Phillips, Steve 
Gorman, Susan Ahmed, Peggy Carr, Sharif Shakrani, 
Arnold Goidstein, Sahar Akhtar, and Maureen Treacy — 
worked closely and collegially with ETS, Westat, and 
NCS staff and played a crucial role in all aspects of the 
program. The NAEP 1994 assessments and reports also 
benefitted from the consistent support and guidance of 
Emerson Elliott, past commissioner of NCES. The 
members of the National Assessment Governing Board 
(NAGB) and the NAGB staff provided invaluable advice 
and guidance throughout. NAEP also owes a debt of 
gratitude to the numerous panelists and consultants 
who provided their expertise and worked so 
conscientiously on developing the assessment. 



The NAEP project at ETS is directed by Paul 
Williams and resides in the Center for the Assessment of 
Educational Progress (CAEP), managed by Archie 
Lapointe and Paul Williams. Steve Lazer managed test 
development activities, and Hilary Persky worked with 
the Geography Item Development committee to develop 
the assessment instruments. Jules Goodison managed 
the operational aspects together with John Olson, and 
sampling and data collection activities were carried out 
by Westat under the direction of Rene Slobasky, Nancy 
Caldwell, and Keith Rust. Printing, distribution, 
scoring, and processing activities were conducted by 
NCS, under the supervision of Judy Moyer, Brad Thayer, 
Mathilde Kennel, Linda Reynolds, and Barbara Price. 

Statistical and psychometric activities for the 
assessments were led by Frank Jenkins under the 
direction of Eugene Johnson, John Mazzeo, and Jim 
Carlson. Edward Kulick performed the geography 
analyses. Karen Miller and John Mazzeo contributed 
substantially to report design activities. Mary Michaels 
and Sharon Davis-Johnson oversaw the production 
aspects, and Loretta Casalaina, James Rura, and 
Roderick Rudder provided further design and 
composition assistance. 

Many thanks are due to the numerous reviewers, 
both internal and external to ETS and NCES. The 
comments and critical feedback provided by the 
following reviewers are reflected in the final version of 
this report: Roger Downs, Robert Duly, James Marran, 
Mary Lyn Bourque, Larry Feinberg, Mary Frase, Gary 
Phillips, Sharif Shakrani, Shi-Chang Wu, Andrew 
Kolstad, Susan Ahmed, Edward Kulick, Karen Miller, 
Nada Ballator, Steve Lazer, and John Mazzeo. Beverly 
Cisney and Alice Kass provided the excellent desktop 
publishing skills essential to the project. 
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